How to create an import content plugin (part 3/4)

This tutorial is part of the « How to create your own plugin »:

Please note: Since we initially published this blog post, we released new versions of Strapi and tutorials may be outdated. Sorry for the inconvenience if it's the case. Please help us by reporting it here.

Table of contents

• Upload From file (part 1)
• Upload from url and raw input (part 2)
• Services (part 3) - current
• History Page (part 4)

You can reach the code for this tutorial at this link

Service: utils

We are going to setup our “analyzing endpoint” so our UI actually receives a proper response when it sends an “analyze request”.

• Go to services directory which is located in plugin’s root directory and create a new directory called utils with a file named utils.js inside it:

  services/utils/utils.js

• Paste the below code in that file:

1"use strict";
2const request = require("request");
3const contentTypeParser = require("content-type-parser");
4const RssParser = require("rss-parser");
5const CsvParser = require("csv-parse/lib/sync");
6const urlRegEx = /((([A-Za-z]{3,9}:(?:\/\/)?)(?:[\-;:&=\+\$,\w]+@)?[A-Za-z0-9\.\-]+|(?:www\.|[\- ;:&=\+\$,\w]+@)[A-Za-z0-9\.\-]+)((?:\/[\+~%\/\.\w\-]*)?\??(?:[\-\+=&;%@\.\w]*)#?(?:[\.\!\/\\\w]*))?)/g;
7const URL_REGEXP = new RegExp(urlRegEx);
8const validateUrl = url => {
9  URL_REGEXP.lastIndex = 0;
10  return URL_REGEXP.test(url);
11};
12const EMAIL_REGEXP = /^(([^<>()\[\]\\.,;:\s@"]+(\.[^<>()\[\]\\.,;:\s@"]+)*)|(".+"))@((\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}])|(([a-zA-Z\-0-9]+\.)+[a-zA-Z]{2,}))$/;
13const stringIsEmail = data => {
14  EMAIL_REGEXP.lastIndex = 0;
15  return EMAIL_REGEXP.test(data);
16};
17const getDataFromUrl = url => {
18  return new Promise((resolve, reject) => {
19    if (!validateUrl(url)) return reject("invalid URL");
20    request(url, null, async (err, res, body) => {
21      if (err) {
22        reject(err);
23      }
24      resolve({ dataType: res.headers["content-type"], body });
25    });
26  });
27};
28const resolveDataFromRequest = async ctx => {
29  const { source, type, options, data } = ctx.request.body;
30  switch (source) {
31    case "upload":
32      return { dataType: type, body: data, options };
33    case "url":
34      const { dataType, body } = await getDataFromUrl(options.url);
35      return { dataType, body, options };
36    case "raw":
37      return {
38        dataType: type,
39        body: options.rawText,
40        options
41      };
42  }
43};

We are about to use above utility functions across our services. resolveDataFromRequest will receive the “analyze request” and detect the type of data alongside its content. For instance, if we are providing “external url” for our source of import, it will use getDataFromUrl function to fetch that url in order to extract the data and its type. Now that we have the data, we must parse that data to extract its items;

• Append the following function to the utils.js:

1const getItemsFromData = ({ dataType, body, options }) =>
2  new Promise(async (resolve, reject) => {
3    const parsedContentType = contentTypeParser(dataType);
4    if (parsedContentType.isXML()) {
5      const parser = new RssParser();
6      const feed = await parser.parseString(body);
7      return resolve({ sourceType: "rss", items: feed.items });
8    }
9    if (dataType === "text/csv" || dataType === "application/vnd.ms-excel") {
10      const items = CsvParser(body, {
11        ...options,
12        columns: true
13      });
14      return resolve({ sourceType: "csv", items });
15    }
16    reject({
17      contentType: parsedContentType.toString()
18    });
19  });

As you can see, we are using different parsers for different types of data. If the data type is xml we are using rss-parser and in case it is csv, we are using csv-parser.

• For the last function, we want to see if a “url” contains any sort of media or not. Append the following code to the utils.js file:

1const urlIsMedia = url => {
2  try {
3    const parsed = new URL(url);
4    const extension = parsed.pathname
5      .split(".")
6      .pop()
7      .toLowerCase();
8    switch (extension) {
9      case "png":
10      case "gif":
11      case "jpg":
12      case "jpeg":
13      case "svg":
14      case "bmp":
15      case "tif":
16      case "tiff":
17        return true;
18      case "mp3":
19      case "wav":
20      case "ogg":
21        return true;
22      case "mp4":
23      case "avi":
24        return true;
25      default:
26        return false;
27    }
28  } catch (error) {
29    return false;
30  }
31};

Well done! The only thing left, is to actually analyze the extracted items.

• Before leaving this file, append the below exports to the bottom of the file:

1module.exports = {
2  resolveDataFromRequest,
3  getItemsFromData,
4  getDataFromUrl,
5  stringIsEmail,
6  urlIsMedia
7};

Service: fieldUtils

• Create a new file called fieldUtils.js inside utils directory:

• services/utils/fieldUtils.js

• Paste the below code inside that file:

1const getUrls = require("get-urls");
2const { urlIsMedia, stringIsEmail } = require("./utils");
3const striptags = require("striptags");
4const detectStringFieldFormat = data => {
5  if (new Date(data).toString() !== "Invalid Date") return "date";
6  if (stringIsEmail(data)) return "email";
7  if (data.length !== striptags(data).length) {
8    return "xml";
9  }
10  return "string";
11};
12const detectFieldFormat = data => {
13  switch (typeof data) {
14    case "number":
15      return "number";
16    case "boolean":
17      return "boolean";
18    case "object":
19      return "object";
20    case "string":
21      return detectStringFieldFormat(data);
22  }
23};

The important factor in our analyzing process is that each item is consisted of several fields and for each field we want to know what exactly is that field’s format. For instance, the above detectFieldFormat function which is doing the format detection for us, when has an encounter with a string field, will do further investigations to find out if that field is a date or a url or even an xml as well.

Beside the field’s format, we want to learn more information such as if the field has a “media url” or what is the “length” of that field;

• Append the following function to the file:

1const compileStatsForFieldData = fieldData => {
2  const stats = {};
3  switch (typeof fieldData) {
4    case "string":
5      try {
6        const urls = Array.from(getUrls(fieldData));
7        const l = urls.length;
8        for (let i = 0; i < l; ++i) {
9          if (urlIsMedia(urls[i])) {
10            stats.hasMediaUrls = true;
11            break;
12          }
13        }
14      } catch (e) {
15        console.log(e);
16      }
17      stats.length = fieldData.length;
18      break;
19    case "object":
20      if (urlIsMedia(fieldData.url)) {
21        stats.hasMediaUrls = true;
22      }
23      stats.length = JSON.stringify(fieldData).length;
24      break;
25    default:
26      console.log(typeof fieldData, fieldData);
27  }
28  stats.format = detectFieldFormat(fieldData);
29  return stats;
30};

• For the last function in this file, we want to extract the “media urls” from the field:

1const getMediaUrlsFromFieldData = fieldData => {
2  switch (typeof fieldData) {
3    case "string":
4      return Array.from(getUrls(fieldData)).filter(urlIsMedia);
5    case "object":
6      return urlIsMedia(fieldData.url) ? [fieldData.url] : [];
7  }
8};

We are all set for the analyzing process!

• Just before leaving the file append the exports at the bottom:

1module.exports = {
2  detectStringFieldFormat,
3  detectFieldFormat,
4  compileStatsForFieldData,
5  getMediaUrlsFromFieldData
6};

Service: analyzer

• Create a new file named analyzer.js inside utils directory:

  services/utils/analyzer.js

• Paste the below code inside that file:

1"use strict";
2const _ = require("lodash");
3var ss = require("simple-statistics");
4const { compileStatsForFieldData } = require("./fieldUtils");
5const getFieldNameSet = items => {
6  const fieldNames = new Set();
7  items.forEach(item => {
8    try {
9      Object.keys(item).forEach(fieldName => fieldNames.add(fieldName));
10    } catch (e) {
11      console.log(e);
12    }
13  });
14  return fieldNames;
15};
16const analyze = (sourceType, items) => {
17  const fieldNames = getFieldNameSet(items);
18  const fieldAnalyses = {};
19  fieldNames.forEach(fieldName => (fieldAnalyses[fieldName] = []));
20  items.forEach(item => {
21    fieldNames.forEach(fieldName => {
22      const fieldData = item[fieldName];
23      const fieldStats = compileStatsForFieldData(fieldData);
24      fieldAnalyses[fieldName].push(fieldStats);
25    });
26  });
27  const fieldStats = Object.keys(fieldAnalyses).map(fieldName => {
28    const fieldAnalysis = fieldAnalyses[fieldName];
29    const fieldStat = { fieldName, count: fieldAnalysis.length };
30    try {
31      fieldStat.format = _.chain(fieldAnalysis)
32        .countBy("format")
33        .map((value, key) => ({ count: value, type: key }))
34        .sortBy("count")
35        .reverse()
36        .head()
37        .get("type")
38        .value();
39    } catch (e) {
40      console.log(e);
41    }
42    fieldStat.hasMediaUrls = fieldAnalysis.some(fa => Boolean(fa.hasMediaUrls));
43    const lengths = _.map(fieldAnalysis, "length");
44    fieldStat.minLength = ss.min(lengths);
45    fieldStat.maxLength = ss.max(lengths);
46    fieldStat.meanLength = ss.mean(lengths).toFixed(2);
47    return fieldStat;
48  });
49  return { itemCount: items.length, fieldStats };
50};
51module.exports = { getFieldNameSet, analyze };

The getFieldNameSet will create a Set of unique field names available in every item. This way we will have all the fields across all the items without repeating ourselves. Then we receive the information about every field by calling compileStatsForFieldData function and based on that information, we will make an educated analytics for each item such as the number of fields per item, the format of fields for every item, existence of any “media url” on the item fields and the calculation of 3 different type of length based on the length of all the fields per item.

Alright! it’s time to actually use our “analyze function”;

• Open ImportContent.js file inside services directory and change its content to the following:

1"use strict";
2/** * ImportContent.js service
3 * * @description: A set of functions similar to controller's actions to avoid code duplication. */
4const { resolveDataFromRequest, getItemsFromData } = require("./utils/utils");
5const analyzer = require("./utils/analyzer");
6module.exports = {
7  preAnalyzeImportFile: async ctx => {
8    const { dataType, body, options } = await resolveDataFromRequest(ctx);
9    const { sourceType, items } = await getItemsFromData({
10      dataType,
11      body,
12      options
13    });
14    const analysis = analyzer.analyze(sourceType, items);
15    return { sourceType, ...analysis };
16  }
17};

Now that we have our service setup, let’s use it inside our controller.

• Open ImportContent.js file inside controllers directory, remove the default content and paste the following:

1"use strict";
2module.exports = {
3  preAnalyzeImportFile: async ctx => {
4    const services = strapi.plugins["import-content"].services;
5    try {
6      const data = await services["importcontent"].preAnalyzeImportFile(ctx);
7      ctx.send(data);
8    } catch (error) {
9      console.log(error);
10      ctx.response.status = 406;
11      ctx.response.message = "could not parse: " + error;
12    }
13  }
14};

Awesome! With an action for the “analyzing process” Let’s setup an end point for this action as well.

• Open routes.json inside config directory and paste the following:

1{
2  "routes": [
3    {
4      "method": "POST",
5      "path": "/preAnalyzeImportFile",
6      "handler": "ImportContent.preAnalyzeImportFile",
7      "config": {
8        "policies": []
9      }
10    }
11  ]
12}

Congratulations, we are a hell of an analyzer!

• Allow public access on our new endpoint, try to upload a “csv file” and watch the console logs when you press the “Analyze” button, as you can see we are receiving the analytics for that file, Awesome!

Now the big question! How do we fill the content type with our imported data? Of course we can restrict the imported data to have the exact fields of our target content type, but this would be a huge disappointment for our plugin; because most of the times, our data will have different field counts and different field names in comparison to our content types; that being said, it is not always the field value that we are looking for, it might be a url and we may be interested in the media that the url is referring to, not the url itself. So what is the solution here? Yes, your guess is correct! We have to provide our users with a way to relate their imported data fields with the content type fields. And that is what we are going to do next

MappingTable Component

• Go to the components directory and create a new directory named MappingTable with an index.js file inside:

  admin/src/components/MappingTable/index.js

• Open the index.js file and paste the following:

1import React, { Component } from "react";
2import PropTypes from "prop-types";
3class MappingTable extends Component {
4  state = { mapping: {} };
5}
6MappingTable.propTypes = {
7  analysis: PropTypes.object.isRequired,
8  targetModel: PropTypes.object,
9  onChange: PropTypes.func
10};
11export default MappingTable;

Before we write our MappingTable UI, there are 2 components that we want to use inside this component.

• Create a new file named TargetFieldSelect.js inside MappingTable directory:

  admin/src/components/MappingTable/TargetFieldSelect.js

• Paste the following inside that file:

1import React, { Component } from "react";
2import { Select } from "@buffetjs/core";
3import { get } from "lodash";
4
5class TargetFieldSelect extends Component {
6  state = {
7    selectedTarget: ""
8  };
9  componentDidMount() {
10    const options = this.fillOptions();
11    this.setState({ selectedTarget: options && options[0] });
12  }
13  onChange(selectedTarget) {
14    this.props.onChange(selectedTarget);
15    this.setState({ selectedTarget });
16  }
17  fillOptions() {
18    const { targetModel } = this.props;
19    const schemaAttributes = get(targetModel, ["schema", "attributes"], {});
20    const options = Object.keys(schemaAttributes)
21      .map(fieldName => {
22        const attribute = get(schemaAttributes, [fieldName], {});
23
24        return attribute.type && { label: fieldName, value: fieldName };
25      })
26      .filter(obj => obj !== undefined);
27
28    return [{ label: "None", value: "none" }, ...options];
29  }
30  render() {
31    return (
32      <Select
33        name={"targetField"}
34        value={this.state.selectedTarget}
35        options={this.fillOptions()}
36        onChange={({ target: { value } }) => this.onChange(value)}
37      />
38    );
39  }
40}
41export default TargetFieldSelect;

This component will show a “Select” element for choosing the desired field among “target content type" fields. This way our users can finally map fields of imported data with fields of their “target content type”. While this is sufficient for most cases, there are scenarios that we want to map certain qualities of a field to our “target content type” fields and not the direct value itself; for instance, if a field is pointing to a media url, we do not want the value of the “url”, instead we want the referenced media itself; or if the field contains any xml, do we want the contents with xml tags or without them (stripe tags).

So we have to show our TargetFieldSelect once again for covering these scenarios. We consider these scenarios as “mapping options”.

• Create a new file named MappingOptions.js inside MappingTable directory:

  admin/src/components/MappingTable/MappingOptions.js

• Paste the below code inside:

1import React, { Component } from "react";
2import TargetFieldSelect from "./TargetFieldSelect";
3import { Label } from "@buffetjs/core";
4const MappingOptions = ({ stat, onChange, targetModel }) => {
5  return (
6    <div>
7      {stat.format === "xml" && (
8        <div>
9          <Label htmlFor={"stripCheckbox"} message={"Strip Tags"} />
10          <input
11            name={"stripCheckbox"}
12            type="checkbox"
13            onChange={e => onChange({ stripTags: e.target.checked })}
14          />
15        </div>
16      )}
17      {stat.hasMediaUrls && (
18        <div style={{ paddingTop: 8, paddingBottom: 8 }}>
19          <Label
20            htmlFor={"mediaTargetSelect"}
21            message={"Import Media to Field"}
22          />
23          <TargetFieldSelect
24            name={"mediaTargetSelect"}
25            targetModel={targetModel}
26            onChange={targetField =>
27              onChange({ importMediaToField: targetField })
28            }
29          />
30        </div>
31      )}
32    </div>
33  );
34};
35export default MappingOptions;

Go back to index.js file inside MappingTable directory to use our newly created components. As you can see we have a mapping object defined in our state that we expect to get filled based on user specified field mappings.

We also expect our HomePage to provide this component with the analysis object which we received from our backend in the previous steps, the “target content type” which has been selected by the user and an onChange function which we will call whenever the user changes our “field mappings”.

• Add the following imports to the index.js file:

1...
2import MappingOptions from "./MappingOptions";
3import TargetFieldSelect from "./TargetFieldSelect";
4import _ from "lodash";
5import Row from "../Row";
6import { Table } from "@buffetjs/core";
7import {
8  Bool as BoolIcon,
9  Json as JsonIcon,
10  Text as TextIcon,
11  NumberIcon,
12  Email as EmailIcon,
13  Calendar as DateIcon,
14  RichText as XmlIcon
15} from "@buffetjs/icons";
16...

For our table, we are going to use “Buffet js” table with a custom row which is filled with the analysis object handed down from HomePage component.

• Paste the following inside MappingTable class:

1CustomRow = ({ row }) => {
2    const { fieldName, count, format, minLength, maxLength, meanLength } = row;
3    return (
4      <tr style={{ paddingTop: 18 }}>
5        <td>{fieldName}</td>
6        <td>
7          <p>{count}</p>
8        </td>
9        <td>
10          {format === "string" && <TextIcon fill="#fdd835" />}
11          {format === "number" && <NumberIcon fill="#fdd835" />}
12          {format === "boolean" && <BoolIcon fill="#fdd835" />}
13          {format === "object" && <JsonIcon fill="#fdd835" />}
14          {format === "email" && <EmailIcon fill="#fdd835" />}
15          {format === "date" && <DateIcon fill="#fdd835" />}
16          {format === "xml" && <XmlIcon fill="#fdd835" />} <p>{format}</p>
17        </td>
18        <td>
19          <span>{minLength}</span>
20        </td>
21        <td>
22          <p>{maxLength}</p>
23        </td>
24        <td>
25          <p>{meanLength}</p>
26        </td>
27        <td>
28          <MappingOptions
29            targetModel={this.props.targetModel}
30            stat={row}
31            onChange={this.changeMappingOptions(row)}
32          />
33        </td>
34        <td>
35          {this.props.targetModel && (
36            <TargetFieldSelect
37              targetModel={this.props.targetModel}
38              onChange={targetField => this.setMapping(fieldName, targetField)}
39            />
40          )}
41        </td>
42      </tr>
43    );
44  };

As you can see, we are using 2 methods named setMapping and changeMappingOptions which are as below:

1  changeMappingOptions = stat => options => {
2    let newState = _.cloneDeep(this.state);
3    for (let key in options) {
4      _.set(newState, `mapping[${stat.fieldName}][${key}]`, options[key]);
5    }
6    this.setState(newState, () => this.props.onChange(this.state.mapping));
7  };
8  setMapping = (source, targetField) => {
9    const state = _.set(
10      this.state,
11      `mapping[${source}]['targetField']`,
12      targetField
13    );
14    this.setState(state, () => this.props.onChange(this.state.mapping));
15    console.log(this.state.mapping);
16  };

What we expect from above methods, is to fill our mapping object based on user specified relations. changeMappingOptions is going to fill the mapping object with the specified options like this:

1{
2  sourceField1: {
3    importMediaToField: targetField1
4  },
5  sourceField2: {
6    stripTags: true
7  }
8}

and setMapping is going to fill the mapping object like below:

1{
2  sourceField1: {
3    targetField: targetField1
4  },
5  sourceField2: {
6    targetField: targetField2
7  }
8}

• Awesome! for the final piece, let’s write our render method:

1  render() {
2    const { analysis } = this.props;
3    const props = {
4      title: "Field Mapping",
5      subtitle:
6        "Configure the Relationship between CSV Fields and Content type Fields"
7    };
8    const headers = [
9      { name: "Field", value: "fieldName" },
10      { name: "Count", value: "count" },
11      { name: "Format", value: "format" },
12      { name: "Min Length", value: "minLength" },
13      { name: "Max Length", value: "maxLength" },
14      { name: "Mean Length", value: "meanLength" },
15      { name: "Options", value: "options" },
16      { name: "Destination", value: "destination" }
17    ];
18    const items = [...analysis.fieldStats];
19    return (
20      <Table
21        {...props}
22        headers={headers}
23        rows={items}
24        customRow={this.CustomRow}
25      />
26    );
27  }

We are all set! Let’s use our MappingTable inside HomePage component.

• Open index.js file at HomePage directory and append the following to the render method output as the last child of parent div (parent div is the tag with container-fluid as its className):

1       {this.state.analysis && (
2          <Row className="row">
3            <MappingTable
4              analysis={this.state.analysis}
5              targetModel={this.getTargetModel()}
6              onChange={this.setFieldMapping}
7            />
8          </Row>
9        )}
10        

• Import MappingTable by adding the following line:

  import MappingTable from "../../components/MappingTable";

• Append the following to index.js file as well:

1  state = {
2    loading: true,
3    modelOptions: [],
4    models: [],
5    importSource: "upload",
6    analyzing: false,
7    analysis: null,
8    selectedContentType: "",
9    fieldMapping: {} // <---
10  };
11
12  getTargetModel = () => { // <---
13    const { models } = this.state;
14    if (!models) return null;
15    return models.find(model => model.uid === this.state.selectedContentType);
16  };
17  
18  setFieldMapping = fieldMapping => { // <---
19    this.setState({ fieldMapping });
20  };

Your index.js file should look like this:

1import React, { memo, Component } from "react";
2import {request} from "strapi-helper-plugin";
3import PropTypes from "prop-types";
4import pluginId from "../../pluginId";
5import UploadFileForm from "../../components/UploadFileForm";
6import ExternalUrlForm from "../../components/ExternalUrlForm";
7import RawInputForm from "../../components/RawInputForm";
8import MappingTable from "../../components/MappingTable";
9
10import {
11  HeaderNav,
12  LoadingIndicator,
13  PluginHeader
14} from "strapi-helper-plugin";
15import Row from "../../components/Row";
16import Block from "../../components/Block";
17import { Select, Label } from "@buffetjs/core";
18import { get, has, isEmpty, pickBy, set } from "lodash";
19
20const getUrl = to =>
21  to ? `/plugins/${pluginId}/${to}` : `/plugins/${pluginId}`;
22
23class HomePage extends Component {
24  importSources = [
25    { label: "External URL ", value: "url" },
26    { label: "Upload file", value: "upload" },
27    { label: "Raw text", value: "raw" }
28  ];
29
30  state = {
31    loading: true,
32    modelOptions: [],
33    models: [],
34    importSource: "upload",
35    analyzing: false,
36    analysis: null,
37    selectedContentType: "",
38    fieldMapping: {}
39  };
40
41  getTargetModel = () => {
42    const { models } = this.state;
43    if (!models) return null;
44    return models.find(model => model.uid === this.state.selectedContentType);
45  };
46  
47  setFieldMapping = fieldMapping => {
48    this.setState({ fieldMapping });
49  };
50
51  selectImportDest = selectedContentType => {
52    this.setState({ selectedContentType });
53  };
54
55  componentDidMount() {
56    this.getModels().then((res) => {
57      const {models, modelOptions} = res
58      this.setState({
59        models,
60        modelOptions,
61        selectedContentType: modelOptions ? modelOptions[0].value : ""
62      });
63    });
64  }
65
66  getModels = async () => {
67    this.setState({ loading: true });
68    try {
69      const response = await request("/content-type-builder/content-types", {
70        method: "GET"
71      });
72
73      // Remove content types from models
74      const models = get(response, ["data"], []).filter(
75        obj => !has(obj, "plugin")
76      );
77      const modelOptions = models.map(model => {
78        return {
79          label: get(model, ["schema", "name"], ""),
80          value: model.uid
81        };
82      });
83
84      this.setState({ loading: false });
85
86      return { models, modelOptions };
87    } catch (e) {
88      this.setState({ loading: false }, () => {
89        strapi.notification.error(`${e}`);
90      });
91    }
92    return [];
93  };
94
95  selectImportSource = importSource => {
96    this.setState({ importSource });
97  };
98
99  onRequestAnalysis = async analysisConfig => {
100    this.analysisConfig = analysisConfig;
101    this.setState({ analyzing: true }, async () => {
102      try {
103        const response = await request("/import-content/preAnalyzeImportFile", {
104          method: "POST",
105          body: analysisConfig
106        });
107
108        this.setState({ analysis: response, analyzing: false }, () => {
109          strapi.notification.success(`Analyzed Successfully`);
110        });
111      } catch (e) {
112        this.setState({ analyzing: false }, () => {
113          strapi.notification.error(`${e}`);
114        });
115      }
116    });
117  };
118
119  render() {
120    return (
121      <div className={"container-fluid"} style={{ padding: "18px 30px" }}>
122        <PluginHeader
123          title={"Import Content"}
124          description={"Import CSV and RSS-Feed into your Content Types"}
125        />
126        <HeaderNav
127          links={[
128            {
129              name: "Import Data",
130              to: getUrl("")
131            },
132            {
133              name: "Import History",
134              to: getUrl("history")
135            }
136          ]}
137          style={{ marginTop: "4.4rem" }}
138        />
139        <div className="row">
140          <Block
141            title="General"
142            description="Configure the Import Source & Destination"
143            style={{ marginBottom: 12 }}
144          >
145            <Row className={"row"}>
146              <div className={"col-4"}>
147                <Label htmlFor="importSource">Import Source</Label>
148                <Select
149                  name="importSource"
150                  options={this.importSources}
151                  value={this.state.importSource}
152                  onChange={({ target: { value } }) =>
153                    this.selectImportSource(value)
154                  }
155                />
156              </div>
157              <div className={"col-4"}>
158                <Label htmlFor="importDest">Import Destination</Label>
159                <Select
160                  value={this.state.selectedContentType}
161                  name="importDest"
162                  options={this.state.modelOptions}
163                  onChange={({ target: { value } }) =>
164                    this.selectImportDest(value)
165                  }
166                />
167              </div>
168            </Row>
169            <Row>
170              {this.state.importSource === "upload" && (
171                <UploadFileForm
172                  onRequestAnalysis={this.onRequestAnalysis}
173                  loadingAnalysis={this.state.analyzing}
174                />
175              )}
176              {this.state.importSource === "url" && (
177                <ExternalUrlForm
178                  onRequestAnalysis={this.onRequestAnalysis}
179                  loadingAnalysis={this.state.analyzing}
180                />
181              )}
182              {this.state.importSource === "raw" && (
183                <RawInputForm
184                  onRequestAnalysis={this.onRequestAnalysis}
185                  loadingAnalysis={this.state.analyzing}
186                />
187              )}
188            </Row>
189          </Block>
190        </div>
191        {this.state.analysis && (
192          <Row className="row">
193            <MappingTable
194              analysis={this.state.analysis}
195              targetModel={this.getTargetModel()}
196              onChange={this.setFieldMapping}
197            />
198          </Row>
199        )}
200      </div>
201    );
202  }
203}
204export default memo(HomePage);

• Import something, analyze and see the result yourself!

Congratulations! the last thing is to add an “Import Button” at the bottom of our MappingTable component to actually “Run the Import“ for us

• Add the Button just under your MappingTable component tag inside HomePage/index.js file as in below code and don't forget to import the Button component at the top of the file:

1import { Button } from "@buffetjs/core";
2
3...
4
5<Button
6  style={{ marginTop: 12 }}
7  label={"Run the Import"}
8  onClick={this.onSaveImport}
9/>
10...

• Add the onSaveImport method as below just under your state:

1onSaveImport = async () => {
2    const { selectedContentType, fieldMapping } = this.state;
3    const { analysisConfig } = this;
4    const importConfig = {
5      ...analysisConfig,
6      contentType: selectedContentType,
7      fieldMapping
8    };
9    try {
10      await request("/import-content", { method: "POST", body: importConfig });
11      this.setState({ saving: false }, () => {
12      strapi.notification.info("Import started");
13      });
14    } catch (e) {
15      strapi.notification.error(`${e}`);
16    }
17  };

Awesome! Whenever the user clicks on “Run the Import” button, we send the “analysis result” alongside “target content type” and user provided “field mapping” to our backend API, and then our import will be initiated

Service: importItems

Let’s setup our endpoint.

• Go to services directory and replace ImportContent.js file with the following:

1"use strict";
2/** * ImportContent.js service
3 * * @description: A set of functions similar to controller's actions to avoid code duplication. */
4const { resolveDataFromRequest, getItemsFromData } = require("./utils/utils");
5const analyzer = require("./utils/analyzer");
6const _ = require("lodash");
7const importFields = require("./utils/importFields");
8const importMediaFiles = require("./utils/importMediaFiles");
9
10const import_queue = {};
11
12module.exports = {
13  preAnalyzeImportFile: async ctx => {
14    const { dataType, body, options } = await resolveDataFromRequest(ctx);
15    const { sourceType, items } = await getItemsFromData({
16      dataType,
17      body,
18      options
19    });
20    const analysis = analyzer.analyze(sourceType, items);
21    return { sourceType, ...analysis };
22  },
23  importItems: (importConfig, ctx) =>
24    new Promise(async (resolve, reject) => {
25      const { dataType, body } = await resolveDataFromRequest(ctx);
26      console.log("importitems", importConfig);
27      try {
28        const { items } = await getItemsFromData({
29          dataType,
30          body,
31          options: importConfig.options
32        });
33        import_queue[importConfig.id] = items;
34      } catch (error) {
35        reject(error);
36      }
37      resolve({
38        status: "import started",
39        importConfigId: importConfig.id
40      });
41      importNextItem(importConfig);
42    }),
43};

Similar to what we did in “analyzing”, we are extracting the data and parsing its items once again. Then we will queue those parsed items for further processing. What we need here is a function that iterates on that queue and stores every item.

• Add the following function before the module.exports:

1const importNextItem = async importConfig => {
2  const sourceItem = import_queue[importConfig.id].shift();
3  if (!sourceItem) {
4    console.log("import complete");
5    await strapi
6      .query("importconfig", "import-content")
7      .update({ id: importConfig.id }, { ongoing: false });
8    return;
9  }
10  try {
11    const importedItem = await importFields(
12      sourceItem,
13      importConfig.fieldMapping
14    );
15    const savedContent = await strapi
16      .query(importConfig.contentType)
17      .create(importedItem);
18    const uploadedFiles = await importMediaFiles(
19      savedContent,
20      sourceItem,
21      importConfig
22    );
23    const fileIds = _.map(_.flatten(uploadedFiles), "id");
24    await strapi.query("importeditem", "import-content").create({
25      importconfig: importConfig.id,
26      ContentId: savedContent.id,
27      ContentType: importConfig.contentType,
28      importedFiles: { fileIds }
29    });
30  } catch (e) {
31    console.log(e);
32  }
33  const { IMPORT_THROTTLE } = strapi.plugins["import-content"].config;
34  setTimeout(() => importNextItem(importConfig), IMPORT_THROTTLE);
35};

Our importing process at above code is consisted of creating the content type, importing the media files, and creating an “imported item” entry that will keep track of our created content types for the future; for instance, if we ever want to undo our imports, we must know which content types entries are related to this specific import.

Service: importFields

Before we continue on to create our action, there are 2 functions that we are calling here: importFields and importMediaFiles.

• Go to utils directory and create a new file named importFields.js:

  services/utils/importFields.js

• Paste the below code inside the new file:

1const striptags = require("striptags");
2const importFields = async (sourceItem, fieldMapping) => {
3  const importedItem = {};
4  Object.keys(fieldMapping).forEach(async sourceField => {
5    const { targetField, stripTags } = fieldMapping[sourceField];
6    if (!targetField || targetField === "none") {
7      return;
8    }
9    const originalValue = sourceItem[sourceField];
10    importedItem[targetField] = stripTags
11      ? striptags(originalValue)
12      : originalValue;
13  });
14  return importedItem;
15};
16module.exports = importFields;

the “field mapping” object is a relation mapping between “imported data item” fields and “target content type” fields. The above function extracts the value for the target fields and generate the candidate object for our “content type” entry on top of those fields.

Service: importMediaFiles

• Create a new file named importMediaFiles.js in utils directory:

  services/utils/importMediaFiles.js

• Paste the following:

1const _ = require("lodash");
2const request = require("request");
3const fileFromBuffer = require("./fileFromBuffer");
4const { getMediaUrlsFromFieldData } = require("../utils/fieldUtils");
5
6const fetchFiles = url =>
7  new Promise((resolve, reject) => {
8    request({ url, method: "GET", encoding: null }, async (err, res, body) => {
9      if (err) {
10        reject(err);
11      }
12      const mimeType = res.headers["content-type"].split(";").shift();
13      const parsed = new URL(url);
14      const extension = parsed.pathname
15        .split(".")
16        .pop()
17        .toLowerCase();
18      resolve(fileFromBuffer(mimeType, extension, body));
19    });
20  });
21const storeFiles = async file => {
22  const uploadProviderConfig = await strapi
23    .store({
24      environment: strapi.config.environment,
25      type: "plugin",
26      name: "upload"
27    })
28    .get({ key: "provider" });
29  return await strapi.plugins["upload"].services["upload"].upload(
30    [file],
31    uploadProviderConfig
32  );
33};
34const relateFileToContent = ({
35  contentType,
36  contentId,
37  targetField,
38  fileBuffer
39}) => {
40  fileBuffer.related = [
41    {
42      refId: contentId,
43      ref: contentType,
44      source: "content-manager",
45      field: targetField
46    }
47  ];
48  return fileBuffer;
49};
50const importMediaFiles = async (savedContent, sourceItem, importConfig) => {
51  const { fieldMapping, contentType } = importConfig;
52  const uploadedFileDescriptors = _.mapValues(
53    fieldMapping,
54    async (mapping, sourceField) => {
55      if (mapping.importMediaToField) {
56        const urls = getMediaUrlsFromFieldData(sourceItem[sourceField]);
57        const fetchPromises = _.uniq(urls).map(fetchFiles);
58        const fileBuffers = await Promise.all(fetchPromises);
59        const relatedContents = fileBuffers.map(fileBuffer =>
60          relateFileToContent({
61            contentType,
62            contentId: savedContent.id,
63            targetField: mapping.importMediaToField,
64            fileBuffer
65          })
66        );
67        const storePromises = relatedContents.map(storeFiles);
68        const storedFiles = await Promise.all(storePromises);
69        console.log(_.flatten(storedFiles));
70        return storedFiles;
71      }
72    }
73  );
74  return await Promise.all(_.values(uploadedFileDescriptors));
75};
76module.exports = importMediaFiles;

What we are doing here is to check if we have importMediaToField specified on the mapping or not, and in case we have, we first extract the urls, then fetch the files for those urls, relate them to the target content type and store them as well.

Service: fileFromBuffer

• Create a new file named fileFromBuffer.js in utils directory:

  services/utils/fileFromBuffer.js

• Paste the following code inside that file:

1const crypto = require('crypto')
2const uuid = require("uuid/v4");
3
4function niceHash(buffer) {
5  return crypto
6    .createHash("sha256")
7    .update(buffer)
8    .digest("base64")
9    .replace(/=/g, "")
10    .replace(/\//g, "-")
11    .replace(/\+/, "_");
12}
13const fileFromBuffer = (mimeType, extension, buffer) => {
14  const fid = uuid();
15  return {
16    buffer,
17    sha256: niceHash(buffer),
18    hash: fid.replace(/-/g, ""),
19    name: `${fid}.${extension}`,
20    ext: `.${extension}`,
21    mime: mimeType,
22    size: (buffer.length / 1000).toFixed(2)
23  };
24};
25module.exports = fileFromBuffer;

our service is ready!

• Open ImportContent.js file inside controllers directory and add the following function:

1create: async ctx => {
2    const services = strapi.plugins["import-content"].services;
3    const importConfig = ctx.request.body;
4    importConfig.ongoing = true;
5    const record = await strapi
6      .query("importconfig", "import-content")
7      .create(importConfig);
8    console.log("create", record);
9    await services["importcontent"].importItems(record, ctx);
10    ctx.send(record);
11}

For each import, we are storing a specific config entry called “import config”. This will help us keep details of our import like the date of this import and the identification of all the content types entries that are created because of this import.

• Open routes.json inside config directory and define our new route:

1{
2      "method": "POST",
3      "path": "/",
4      "handler": "ImportContent.create",
5      "config": {
6        "policies": []
7      }
8}

Awesome! Our import endpoint is implemented; The main purpose of this plugin is accomplished. The rest is to show our initiated imports to the users. Also we are going to allow our users to “undo” the initiated imports as well as “removing” their related configs.

• Add the following function to ImportContent.js file inside controllers directory:

1index: async ctx => {
2    const entries = await strapi.query("importconfig", "import-content").find();
3    const withCounts = entries.map(entry => ({
4      ...entry,
5      importedCount: entry.importeditems.length,
6      importeditems: []
7    }));
8    const withName = withCounts.map(entry =>
9      ({
10        ...entry,
11        contentType: strapi.contentTypes[entry.contentType].info.name || 
12        entry.contentType
13      }))
14    ctx.send(withName);
15}

it will retrieve the stored configs for user initiated imports. This will allow us to provide our users with a history of their imports and show some details about each import. Beside retrieving the import configs, it would be nice to have a way to delete them as well.

• Append the following function to the file:

1  delete: async ctx => {
2    const importId = ctx.params.importId;
3    const res = await strapi.query("importconfig", "import-content").delete({
4      id: importId
5    });
6    if (res && res.id) {
7      ctx.send(res.id);
8    } else {
9      ctx.response.status = 400;
10      ctx.response.message = "could not delete: the provided id might be wrong";
11    }
12  }

The only thing left is a way to undo our imports. For this we are about to implement a service similar to what we did for importing data.

Service: undoImports

• Open ImportContent.js file in services directory, and add the following before the module.exports:

1const undo_queue = {};
2const removeImportedFiles = async (fileIds, uploadConfig) => {
3  const removePromises = fileIds.map(id =>
4    strapi.plugins["upload"].services.upload.remove({ id }, uploadConfig)
5  );
6  return await Promise.all(removePromises);
7};
8const undoNextItem = async (importConfig, uploadConfig) => {
9  const item = undo_queue[importConfig.id].shift();
10  if (!item) {
11    console.log("undo complete");
12    await strapi
13      .query("importconfig", "import-content")
14      .update({ id: importConfig.id }, { ongoing: false });
15    return;
16  }
17  try {
18    await strapi.query(importConfig.contentType).delete({ id: item.ContentId });
19  } catch (e) {
20    console.log(e);
21  }
22  try {
23    const importedFileIds = _.compact(item.importedFiles.fileIds);
24    await removeImportedFiles(importedFileIds, uploadConfig);
25  } catch (e) {
26    console.log(e);
27  }
28  try {
29    await strapi.query("importeditem", "import-content").delete({
30      id: item.id
31    });
32  } catch (e) {
33    console.log(e);
34  }
35  const { UNDO_THROTTLE } = strapi.plugins["import-content"].config;
36  setTimeout(() => undoNextItem(importConfig, uploadConfig), UNDO_THROTTLE);
37};

The undoNextItem function is iterating on the queue for removing every imported item. This undo process is consisted of removing the content type entry, removing the related files and the imported item entry as well.

• Now add the below function to module.exports:

1undoItems: importConfig =>
2    new Promise(async (resolve, reject) => {
3      try {
4        undo_queue[importConfig.id] = importConfig.importeditems;
5      } catch (error) {
6        reject(error);
7      }
8      await strapi
9        .query("importconfig", "import-content")
10        .update({ id: importConfig.id }, { ongoing: true });
11      resolve({
12        status: "undo started",
13        importConfigId: importConfig.id
14      });
15      const uploadConfig = await strapi
16        .store({
17          environment: strapi.config.environment,
18          type: "plugin",
19          name: "upload"
20        })
21        .get({ key: "provider" });
22      undoNextItem(importConfig, uploadConfig);
23}),

well done!

• Go back to ImportContent.js file inside controllers directory and add the following function:

1undo: async ctx => {
2    const services = strapi.plugins["import-content"].services;
3    const importId = ctx.params.importId;
4    const importConfig = await strapi
5      .query("importconfig", "import-content")
6      .findOne({ id: importId });
7    console.log("undo", importId);
8    await services["importcontent"].undoItems(importConfig);
9    ctx.send(importConfig);
10}

• Open routes.json file at config directory and append the following routes:

1    {
2      "method": "GET",
3      "path": "/",
4      "handler": "ImportContent.index",
5      "config": {
6        "policies": []
7      }
8    },
9    {
10      "method": "DELETE",
11      "path": "/:importId",
12      "handler": "ImportContent.delete",
13      "config": {
14        "policies": []
15      }
16    },
17    {
18      "method": "POST",
19      "path": "/:importId/undo",
20      "handler": "ImportContent.undo",
21      "config": {
22        "policies": []
23      }
24    }

Now that we have all the endpoints we need, let’s go back to our admin directory and implement the HistoryPage. This page will show a history of all the user initiated imports, their status and 2 buttons for undo and delete actions.

Warning! while we are OK to continue to the next section, there is one point we have to mention here: Until now we have only provided endpoints for the functionalities that we already have like "analyzing", "importing" and running an "undo" operation. The point is that if we want to be consistent with other plugins like "graphql", we must define a set of controllers for each one of our content types which in our case are importconfig & importeditem. These controllers are the famous set of find, findOne, count, update and delete which i bet you are already familiar with them. So in case that you want this plugin to be efficient in production environments, you must make it consistent. Before jumping to the next section, please do the followings:

• Create a new file named ImportConfig in controllers directory:

controllers/ImportConfig.js

• Paste the following code inside that file:

1module.exports = {
2  count: async ctx => {
3    const entries = await strapi.query("importconfig", "import-content").count(ctx.request.query)
4    ctx.send(entries)
5  },
6  findOne: async ctx => {
7    const entries = await strapi.query("importconfig", "import-content").findOne({id: ctx.params.importId})
8    ctx.send(entries)
9  },
10  find: async ctx => {
11    const entries = await strapi.query("importconfig", "import-content").find(ctx.request.query)
12    ctx.send(entries)
13  },
14  delete: async ctx => {
15    return strapi.query("importconfig", "import-content").delete({id: ctx.params.importId})
16  },
17  update: async ctx => {
18    return strapi.query("importconfig", "import-content").update({id: ctx.params.importId}, ctx.request.body)
19  }
20};

• Create a new file named ImportedItem in controllers directory:

controllers/ImportedItem.js

• Paste the following code inside that file:

1module.exports = {
2  count: async ctx => {
3    const entries = await strapi.query("importeditem", "import-content").count(ctx.request.query)
4    ctx.send(entries)
5  },
6  findOne: async ctx => {
7    const entries = await strapi.query("importeditem", "import-content").findOne({id: ctx.params.importId})
8    ctx.send(entries)
9  },
10  find: async ctx => {
11    const entries = await strapi.query("importeditem", "import-content").find(ctx.request.query)
12    ctx.send(entries)
13  },
14  delete: async ctx => {
15    return strapi.query("importeditem", "import-content").delete({id: ctx.params.importId})
16  },
17  update: async ctx => {
18    return strapi.query("importeditem", "import-content").update({id: ctx.params.importId}, ctx.request.body)
19  }
20};

• For the last step, add the below routes to the routes.json:

config/routes.json

1{
2  "method": "GET",
3  "path": "/importconfig",
4  "handler": "ImportConfig.find",
5  "config": {
6    "policies": []
7  }
8},
9{
10  "method": "GET",
11  "path": "/importconfig/:importId",
12  "handler": "ImportConfig.findOne",
13  "config": {
14    "policies": []
15  }
16},
17{
18  "method": "GET",
19  "path": "/importconfig/count",
20  "handler": "ImportConfig.count",
21  "config": {
22    "policies": []
23  }
24},
25{
26  "method": "PUT",
27  "path": "/importconfig/:importId",
28  "handler": "ImportConfig.update",
29  "config": {
30    "policies": []
31  }
32},
33{
34  "method": "DELETE",
35  "path": "/importconfig/:importId",
36  "handler": "ImportConfig.delete",
37  "config": {
38    "policies": []
39  }
40},
41{
42  "method": "GET",
43  "path": "/importeditem",
44  "handler": "ImportedItem.find",
45  "config": {
46    "policies": []
47  }
48},
49{
50  "method": "GET",
51  "path": "/importeditem/:importId",
52  "handler": "ImportedItem.findOne",
53  "config": {
54    "policies": []
55  }
56},
57{
58  "method": "GET",
59  "path": "/importeditem/count",
60  "handler": "ImportedItem.count",
61  "config": {
62    "policies": []
63  }
64},
65{
66  "method": "PUT",
67  "path": "/importeditem/:importId",
68  "handler": "ImportedItem.update",
69  "config": {
70    "policies": []
71  }
72},
73{
74  "method": "DELETE",
75  "path": "/importeditem/:importId",
76  "handler": "ImportedItem.delete",
77  "config": {
78    "policies": []
79  }
80}

Great you finished the third part of this tutorial! Let's create the History page! https://strapi.io/blog/how-to-create-an-import-content-plugin-part-4-4