This tutorial is part of the « How to create your own plugin »:
Please note: Since we initially published this blog post, we released new versions of Strapi and tutorials may be outdated. Sorry for the inconvenience if it's the case. Please help us by reporting it here.
Table of contents
You can reach the code for this tutorial at this link
We are going to setup our “analyzing endpoint” so our UI actually receives a proper response when it sends an “analyze request”.
Go to services
directory which is located in plugin’s root directory and create a new directory called utils
with a file named utils.js
inside it:
services/utils/utils.js
Paste the below code in that file:
1"use strict";
2const request = require("request");
3const contentTypeParser = require("content-type-parser");
4const RssParser = require("rss-parser");
5const CsvParser = require("csv-parse/lib/sync");
6const urlRegEx = /((([A-Za-z]{3,9}:(?:\/\/)?)(?:[\-;:&=\+\$,\w]+@)?[A-Za-z0-9\.\-]+|(?:www\.|[\- ;:&=\+\$,\w]+@)[A-Za-z0-9\.\-]+)((?:\/[\+~%\/\.\w\-]*)?\??(?:[\-\+=&;%@\.\w]*)#?(?:[\.\!\/\\\w]*))?)/g;
7const URL_REGEXP = new RegExp(urlRegEx);
8const validateUrl = url => {
9 URL_REGEXP.lastIndex = 0;
10 return URL_REGEXP.test(url);
11};
12const EMAIL_REGEXP = /^(([^<>()\[\]\\.,;:\s@"]+(\.[^<>()\[\]\\.,;:\s@"]+)*)|(".+"))@((\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}])|(([a-zA-Z\-0-9]+\.)+[a-zA-Z]{2,}))$/;
13const stringIsEmail = data => {
14 EMAIL_REGEXP.lastIndex = 0;
15 return EMAIL_REGEXP.test(data);
16};
17const getDataFromUrl = url => {
18 return new Promise((resolve, reject) => {
19 if (!validateUrl(url)) return reject("invalid URL");
20 request(url, null, async (err, res, body) => {
21 if (err) {
22 reject(err);
23 }
24 resolve({ dataType: res.headers["content-type"], body });
25 });
26 });
27};
28const resolveDataFromRequest = async ctx => {
29 const { source, type, options, data } = ctx.request.body;
30 switch (source) {
31 case "upload":
32 return { dataType: type, body: data, options };
33 case "url":
34 const { dataType, body } = await getDataFromUrl(options.url);
35 return { dataType, body, options };
36 case "raw":
37 return {
38 dataType: type,
39 body: options.rawText,
40 options
41 };
42 }
43};
We are about to use above utility functions across our services. resolveDataFromRequest
will receive the “analyze request” and detect the type of data alongside its content. For instance, if we are providing “external url” for our source of import, it will use getDataFromUrl
function to fetch that url in order to extract the data and its type. Now that we have the data, we must parse that data to extract its items;
utils.js
:1const getItemsFromData = ({ dataType, body, options }) =>
2 new Promise(async (resolve, reject) => {
3 const parsedContentType = contentTypeParser(dataType);
4 if (parsedContentType.isXML()) {
5 const parser = new RssParser();
6 const feed = await parser.parseString(body);
7 return resolve({ sourceType: "rss", items: feed.items });
8 }
9 if (dataType === "text/csv" || dataType === "application/vnd.ms-excel") {
10 const items = CsvParser(body, {
11 ...options,
12 columns: true
13 });
14 return resolve({ sourceType: "csv", items });
15 }
16 reject({
17 contentType: parsedContentType.toString()
18 });
19 });
As you can see, we are using different parsers for different types of data. If the data type is xml
we are using rss-parser
and in case it is csv
, we are using csv-parser
.
utils.js
file:1const urlIsMedia = url => {
2 try {
3 const parsed = new URL(url);
4 const extension = parsed.pathname
5 .split(".")
6 .pop()
7 .toLowerCase();
8 switch (extension) {
9 case "png":
10 case "gif":
11 case "jpg":
12 case "jpeg":
13 case "svg":
14 case "bmp":
15 case "tif":
16 case "tiff":
17 return true;
18 case "mp3":
19 case "wav":
20 case "ogg":
21 return true;
22 case "mp4":
23 case "avi":
24 return true;
25 default:
26 return false;
27 }
28 } catch (error) {
29 return false;
30 }
31};
Well done! The only thing left, is to actually analyze the extracted items.
1module.exports = {
2 resolveDataFromRequest,
3 getItemsFromData,
4 getDataFromUrl,
5 stringIsEmail,
6 urlIsMedia
7};
Create a new file called fieldUtils.js
inside utils
directory:
services/utils/fieldUtils.js
Paste the below code inside that file:
1const getUrls = require("get-urls");
2const { urlIsMedia, stringIsEmail } = require("./utils");
3const striptags = require("striptags");
4const detectStringFieldFormat = data => {
5 if (new Date(data).toString() !== "Invalid Date") return "date";
6 if (stringIsEmail(data)) return "email";
7 if (data.length !== striptags(data).length) {
8 return "xml";
9 }
10 return "string";
11};
12const detectFieldFormat = data => {
13 switch (typeof data) {
14 case "number":
15 return "number";
16 case "boolean":
17 return "boolean";
18 case "object":
19 return "object";
20 case "string":
21 return detectStringFieldFormat(data);
22 }
23};
The important factor in our analyzing process is that each item is consisted of several fields and for each field we want to know what exactly is that field’s format. For instance, the above detectFieldFormat
function which is doing the format detection for us, when has an encounter with a string field, will do further investigations to find out if that field is a date
or a url
or even an xml
as well.
Beside the field’s format, we want to learn more information such as if the field has a “media url” or what is the “length” of that field;
1const compileStatsForFieldData = fieldData => {
2 const stats = {};
3 switch (typeof fieldData) {
4 case "string":
5 try {
6 const urls = Array.from(getUrls(fieldData));
7 const l = urls.length;
8 for (let i = 0; i < l; ++i) {
9 if (urlIsMedia(urls[i])) {
10 stats.hasMediaUrls = true;
11 break;
12 }
13 }
14 } catch (e) {
15 console.log(e);
16 }
17 stats.length = fieldData.length;
18 break;
19 case "object":
20 if (urlIsMedia(fieldData.url)) {
21 stats.hasMediaUrls = true;
22 }
23 stats.length = JSON.stringify(fieldData).length;
24 break;
25 default:
26 console.log(typeof fieldData, fieldData);
27 }
28 stats.format = detectFieldFormat(fieldData);
29 return stats;
30};
1const getMediaUrlsFromFieldData = fieldData => {
2 switch (typeof fieldData) {
3 case "string":
4 return Array.from(getUrls(fieldData)).filter(urlIsMedia);
5 case "object":
6 return urlIsMedia(fieldData.url) ? [fieldData.url] : [];
7 }
8};
We are all set for the analyzing process!
1module.exports = {
2 detectStringFieldFormat,
3 detectFieldFormat,
4 compileStatsForFieldData,
5 getMediaUrlsFromFieldData
6};
Create a new file named analyzer.js
inside utils
directory:
services/utils/analyzer.js
Paste the below code inside that file:
1"use strict";
2const _ = require("lodash");
3var ss = require("simple-statistics");
4const { compileStatsForFieldData } = require("./fieldUtils");
5const getFieldNameSet = items => {
6 const fieldNames = new Set();
7 items.forEach(item => {
8 try {
9 Object.keys(item).forEach(fieldName => fieldNames.add(fieldName));
10 } catch (e) {
11 console.log(e);
12 }
13 });
14 return fieldNames;
15};
16const analyze = (sourceType, items) => {
17 const fieldNames = getFieldNameSet(items);
18 const fieldAnalyses = {};
19 fieldNames.forEach(fieldName => (fieldAnalyses[fieldName] = []));
20 items.forEach(item => {
21 fieldNames.forEach(fieldName => {
22 const fieldData = item[fieldName];
23 const fieldStats = compileStatsForFieldData(fieldData);
24 fieldAnalyses[fieldName].push(fieldStats);
25 });
26 });
27 const fieldStats = Object.keys(fieldAnalyses).map(fieldName => {
28 const fieldAnalysis = fieldAnalyses[fieldName];
29 const fieldStat = { fieldName, count: fieldAnalysis.length };
30 try {
31 fieldStat.format = _.chain(fieldAnalysis)
32 .countBy("format")
33 .map((value, key) => ({ count: value, type: key }))
34 .sortBy("count")
35 .reverse()
36 .head()
37 .get("type")
38 .value();
39 } catch (e) {
40 console.log(e);
41 }
42 fieldStat.hasMediaUrls = fieldAnalysis.some(fa => Boolean(fa.hasMediaUrls));
43 const lengths = _.map(fieldAnalysis, "length");
44 fieldStat.minLength = ss.min(lengths);
45 fieldStat.maxLength = ss.max(lengths);
46 fieldStat.meanLength = ss.mean(lengths).toFixed(2);
47 return fieldStat;
48 });
49 return { itemCount: items.length, fieldStats };
50};
51module.exports = { getFieldNameSet, analyze };
The getFieldNameSet
will create a Set of unique field names available in every item. This way we will have all the fields across all the items without repeating ourselves. Then we receive the information about every field by calling compileStatsForFieldData
function and based on that information, we will make an educated analytics for each item such as the number of fields per item, the format of fields for every item, existence of any “media url” on the item fields and the calculation of 3 different type of length based on the length of all the fields per item.
Alright! it’s time to actually use our “analyze function”;
ImportContent.js
file inside services
directory and change its content to the following:1"use strict";
2/** * ImportContent.js service
3 * * @description: A set of functions similar to controller's actions to avoid code duplication. */
4const { resolveDataFromRequest, getItemsFromData } = require("./utils/utils");
5const analyzer = require("./utils/analyzer");
6module.exports = {
7 preAnalyzeImportFile: async ctx => {
8 const { dataType, body, options } = await resolveDataFromRequest(ctx);
9 const { sourceType, items } = await getItemsFromData({
10 dataType,
11 body,
12 options
13 });
14 const analysis = analyzer.analyze(sourceType, items);
15 return { sourceType, ...analysis };
16 }
17};
Now that we have our service setup, let’s use it inside our controller.
ImportContent.js
file inside controllers
directory, remove the default content and paste the following:1"use strict";
2module.exports = {
3 preAnalyzeImportFile: async ctx => {
4 const services = strapi.plugins["import-content"].services;
5 try {
6 const data = await services["importcontent"].preAnalyzeImportFile(ctx);
7 ctx.send(data);
8 } catch (error) {
9 console.log(error);
10 ctx.response.status = 406;
11 ctx.response.message = "could not parse: " + error;
12 }
13 }
14};
Awesome! With an action for the “analyzing process” Let’s setup an end point for this action as well.
routes.json
inside config
directory and paste the following:1{
2 "routes": [
3 {
4 "method": "POST",
5 "path": "/preAnalyzeImportFile",
6 "handler": "ImportContent.preAnalyzeImportFile",
7 "config": {
8 "policies": []
9 }
10 }
11 ]
12}
Congratulations, we are a hell of an analyzer!
Now the big question! How do we fill the content type with our imported data? Of course we can restrict the imported data to have the exact fields of our target content type, but this would be a huge disappointment for our plugin; because most of the times, our data will have different field counts and different field names in comparison to our content types; that being said, it is not always the field value that we are looking for, it might be a url and we may be interested in the media that the url is referring to, not the url itself. So what is the solution here? Yes, your guess is correct! We have to provide our users with a way to relate their imported data fields with the content type fields. And that is what we are going to do next
Go to the components
directory and create a new directory named MappingTable
with an index.js
file inside:
admin/src/components/MappingTable/index.js
Open the index.js
file and paste the following:
1import React, { Component } from "react";
2import PropTypes from "prop-types";
3class MappingTable extends Component {
4 state = { mapping: {} };
5}
6MappingTable.propTypes = {
7 analysis: PropTypes.object.isRequired,
8 targetModel: PropTypes.object,
9 onChange: PropTypes.func
10};
11export default MappingTable;
Before we write our MappingTable
UI, there are 2 components that we want to use inside this component.
Create a new file named TargetFieldSelect.js
inside MappingTable
directory:
admin/src/components/MappingTable/TargetFieldSelect.js
Paste the following inside that file:
1import React, { Component } from "react";
2import { Select } from "@buffetjs/core";
3import { get } from "lodash";
4
5class TargetFieldSelect extends Component {
6 state = {
7 selectedTarget: ""
8 };
9 componentDidMount() {
10 const options = this.fillOptions();
11 this.setState({ selectedTarget: options && options[0] });
12 }
13 onChange(selectedTarget) {
14 this.props.onChange(selectedTarget);
15 this.setState({ selectedTarget });
16 }
17 fillOptions() {
18 const { targetModel } = this.props;
19 const schemaAttributes = get(targetModel, ["schema", "attributes"], {});
20 const options = Object.keys(schemaAttributes)
21 .map(fieldName => {
22 const attribute = get(schemaAttributes, [fieldName], {});
23
24 return attribute.type && { label: fieldName, value: fieldName };
25 })
26 .filter(obj => obj !== undefined);
27
28 return [{ label: "None", value: "none" }, ...options];
29 }
30 render() {
31 return (
32 <Select
33 name={"targetField"}
34 value={this.state.selectedTarget}
35 options={this.fillOptions()}
36 onChange={({ target: { value } }) => this.onChange(value)}
37 />
38 );
39 }
40}
41export default TargetFieldSelect;
This component will show a “Select” element for choosing the desired field among “target content type" fields. This way our users can finally map fields of imported data with fields of their “target content type”. While this is sufficient for most cases, there are scenarios that we want to map certain qualities of a field to our “target content type” fields and not the direct value itself; for instance, if a field is pointing to a media url, we do not want the value of the “url”, instead we want the referenced media itself; or if the field contains any xml, do we want the contents with xml tags or without them (stripe tags).
So we have to show our TargetFieldSelect
once again for covering these scenarios. We consider these scenarios as “mapping options”.
Create a new file named MappingOptions.js
inside MappingTable
directory:
admin/src/components/MappingTable/MappingOptions.js
Paste the below code inside:
1import React, { Component } from "react";
2import TargetFieldSelect from "./TargetFieldSelect";
3import { Label } from "@buffetjs/core";
4const MappingOptions = ({ stat, onChange, targetModel }) => {
5 return (
6 <div>
7 {stat.format === "xml" && (
8 <div>
9 <Label htmlFor={"stripCheckbox"} message={"Strip Tags"} />
10 <input
11 name={"stripCheckbox"}
12 type="checkbox"
13 onChange={e => onChange({ stripTags: e.target.checked })}
14 />
15 </div>
16 )}
17 {stat.hasMediaUrls && (
18 <div style={{ paddingTop: 8, paddingBottom: 8 }}>
19 <Label
20 htmlFor={"mediaTargetSelect"}
21 message={"Import Media to Field"}
22 />
23 <TargetFieldSelect
24 name={"mediaTargetSelect"}
25 targetModel={targetModel}
26 onChange={targetField =>
27 onChange({ importMediaToField: targetField })
28 }
29 />
30 </div>
31 )}
32 </div>
33 );
34};
35export default MappingOptions;
Go back to index.js
file inside MappingTable
directory to use our newly created components. As you can see we have a mapping
object defined in our state that we expect to get filled based on user specified field mappings.
We also expect our HomePage
to provide this component with the analysis
object which we received from our backend in the previous steps, the “target content type” which has been selected by the user and an onChange
function which we will call whenever the user changes our “field mappings”.
index.js
file:1...
2import MappingOptions from "./MappingOptions";
3import TargetFieldSelect from "./TargetFieldSelect";
4import _ from "lodash";
5import Row from "../Row";
6import { Table } from "@buffetjs/core";
7import {
8 Bool as BoolIcon,
9 Json as JsonIcon,
10 Text as TextIcon,
11 NumberIcon,
12 Email as EmailIcon,
13 Calendar as DateIcon,
14 RichText as XmlIcon
15} from "@buffetjs/icons";
16...
For our table, we are going to use “Buffet js” table with a custom row which is filled with the analysis
object handed down from HomePage
component.
MappingTable
class:1CustomRow = ({ row }) => {
2 const { fieldName, count, format, minLength, maxLength, meanLength } = row;
3 return (
4 <tr style={{ paddingTop: 18 }}>
5 <td>{fieldName}</td>
6 <td>
7 <p>{count}</p>
8 </td>
9 <td>
10 {format === "string" && <TextIcon fill="#fdd835" />}
11 {format === "number" && <NumberIcon fill="#fdd835" />}
12 {format === "boolean" && <BoolIcon fill="#fdd835" />}
13 {format === "object" && <JsonIcon fill="#fdd835" />}
14 {format === "email" && <EmailIcon fill="#fdd835" />}
15 {format === "date" && <DateIcon fill="#fdd835" />}
16 {format === "xml" && <XmlIcon fill="#fdd835" />} <p>{format}</p>
17 </td>
18 <td>
19 <span>{minLength}</span>
20 </td>
21 <td>
22 <p>{maxLength}</p>
23 </td>
24 <td>
25 <p>{meanLength}</p>
26 </td>
27 <td>
28 <MappingOptions
29 targetModel={this.props.targetModel}
30 stat={row}
31 onChange={this.changeMappingOptions(row)}
32 />
33 </td>
34 <td>
35 {this.props.targetModel && (
36 <TargetFieldSelect
37 targetModel={this.props.targetModel}
38 onChange={targetField => this.setMapping(fieldName, targetField)}
39 />
40 )}
41 </td>
42 </tr>
43 );
44 };
As you can see, we are using 2 methods named setMapping
and changeMappingOptions
which are as below:
1 changeMappingOptions = stat => options => {
2 let newState = _.cloneDeep(this.state);
3 for (let key in options) {
4 _.set(newState, `mapping[${stat.fieldName}][${key}]`, options[key]);
5 }
6 this.setState(newState, () => this.props.onChange(this.state.mapping));
7 };
8 setMapping = (source, targetField) => {
9 const state = _.set(
10 this.state,
11 `mapping[${source}]['targetField']`,
12 targetField
13 );
14 this.setState(state, () => this.props.onChange(this.state.mapping));
15 console.log(this.state.mapping);
16 };
What we expect from above methods, is to fill our mapping
object based on user specified relations. changeMappingOptions
is going to fill the mapping
object with the specified options like this:
1{
2 sourceField1: {
3 importMediaToField: targetField1
4 },
5 sourceField2: {
6 stripTags: true
7 }
8}
and setMapping
is going to fill the mapping
object like below:
1{
2 sourceField1: {
3 targetField: targetField1
4 },
5 sourceField2: {
6 targetField: targetField2
7 }
8}
render
method:1 render() {
2 const { analysis } = this.props;
3 const props = {
4 title: "Field Mapping",
5 subtitle:
6 "Configure the Relationship between CSV Fields and Content type Fields"
7 };
8 const headers = [
9 { name: "Field", value: "fieldName" },
10 { name: "Count", value: "count" },
11 { name: "Format", value: "format" },
12 { name: "Min Length", value: "minLength" },
13 { name: "Max Length", value: "maxLength" },
14 { name: "Mean Length", value: "meanLength" },
15 { name: "Options", value: "options" },
16 { name: "Destination", value: "destination" }
17 ];
18 const items = [...analysis.fieldStats];
19 return (
20 <Table
21 {...props}
22 headers={headers}
23 rows={items}
24 customRow={this.CustomRow}
25 />
26 );
27 }
We are all set! Let’s use our MappingTable
inside HomePage
component.
index.js
file at HomePage
directory and append the following to the render
method output as the last child of parent div
(parent div
is the tag with container-fluid
as its className
):1 {this.state.analysis && (
2 <Row className="row">
3 <MappingTable
4 analysis={this.state.analysis}
5 targetModel={this.getTargetModel()}
6 onChange={this.setFieldMapping}
7 />
8 </Row>
9 )}
10
Import MappingTable
by adding the following line:
import MappingTable from "../../components/MappingTable";
Append the following to index.js
file as well:
1 state = {
2 loading: true,
3 modelOptions: [],
4 models: [],
5 importSource: "upload",
6 analyzing: false,
7 analysis: null,
8 selectedContentType: "",
9 fieldMapping: {} // <---
10 };
11
12 getTargetModel = () => { // <---
13 const { models } = this.state;
14 if (!models) return null;
15 return models.find(model => model.uid === this.state.selectedContentType);
16 };
17
18 setFieldMapping = fieldMapping => { // <---
19 this.setState({ fieldMapping });
20 };
Your index.js
file should look like this:
1import React, { memo, Component } from "react";
2import {request} from "strapi-helper-plugin";
3import PropTypes from "prop-types";
4import pluginId from "../../pluginId";
5import UploadFileForm from "../../components/UploadFileForm";
6import ExternalUrlForm from "../../components/ExternalUrlForm";
7import RawInputForm from "../../components/RawInputForm";
8import MappingTable from "../../components/MappingTable";
9
10import {
11 HeaderNav,
12 LoadingIndicator,
13 PluginHeader
14} from "strapi-helper-plugin";
15import Row from "../../components/Row";
16import Block from "../../components/Block";
17import { Select, Label } from "@buffetjs/core";
18import { get, has, isEmpty, pickBy, set } from "lodash";
19
20const getUrl = to =>
21 to ? `/plugins/${pluginId}/${to}` : `/plugins/${pluginId}`;
22
23class HomePage extends Component {
24 importSources = [
25 { label: "External URL ", value: "url" },
26 { label: "Upload file", value: "upload" },
27 { label: "Raw text", value: "raw" }
28 ];
29
30 state = {
31 loading: true,
32 modelOptions: [],
33 models: [],
34 importSource: "upload",
35 analyzing: false,
36 analysis: null,
37 selectedContentType: "",
38 fieldMapping: {}
39 };
40
41 getTargetModel = () => {
42 const { models } = this.state;
43 if (!models) return null;
44 return models.find(model => model.uid === this.state.selectedContentType);
45 };
46
47 setFieldMapping = fieldMapping => {
48 this.setState({ fieldMapping });
49 };
50
51 selectImportDest = selectedContentType => {
52 this.setState({ selectedContentType });
53 };
54
55 componentDidMount() {
56 this.getModels().then((res) => {
57 const {models, modelOptions} = res
58 this.setState({
59 models,
60 modelOptions,
61 selectedContentType: modelOptions ? modelOptions[0].value : ""
62 });
63 });
64 }
65
66 getModels = async () => {
67 this.setState({ loading: true });
68 try {
69 const response = await request("/content-type-builder/content-types", {
70 method: "GET"
71 });
72
73 // Remove content types from models
74 const models = get(response, ["data"], []).filter(
75 obj => !has(obj, "plugin")
76 );
77 const modelOptions = models.map(model => {
78 return {
79 label: get(model, ["schema", "name"], ""),
80 value: model.uid
81 };
82 });
83
84 this.setState({ loading: false });
85
86 return { models, modelOptions };
87 } catch (e) {
88 this.setState({ loading: false }, () => {
89 strapi.notification.error(`${e}`);
90 });
91 }
92 return [];
93 };
94
95 selectImportSource = importSource => {
96 this.setState({ importSource });
97 };
98
99 onRequestAnalysis = async analysisConfig => {
100 this.analysisConfig = analysisConfig;
101 this.setState({ analyzing: true }, async () => {
102 try {
103 const response = await request("/import-content/preAnalyzeImportFile", {
104 method: "POST",
105 body: analysisConfig
106 });
107
108 this.setState({ analysis: response, analyzing: false }, () => {
109 strapi.notification.success(`Analyzed Successfully`);
110 });
111 } catch (e) {
112 this.setState({ analyzing: false }, () => {
113 strapi.notification.error(`${e}`);
114 });
115 }
116 });
117 };
118
119 render() {
120 return (
121 <div className={"container-fluid"} style={{ padding: "18px 30px" }}>
122 <PluginHeader
123 title={"Import Content"}
124 description={"Import CSV and RSS-Feed into your Content Types"}
125 />
126 <HeaderNav
127 links={[
128 {
129 name: "Import Data",
130 to: getUrl("")
131 },
132 {
133 name: "Import History",
134 to: getUrl("history")
135 }
136 ]}
137 style={{ marginTop: "4.4rem" }}
138 />
139 <div className="row">
140 <Block
141 title="General"
142 description="Configure the Import Source & Destination"
143 style={{ marginBottom: 12 }}
144 >
145 <Row className={"row"}>
146 <div className={"col-4"}>
147 <Label htmlFor="importSource">Import Source</Label>
148 <Select
149 name="importSource"
150 options={this.importSources}
151 value={this.state.importSource}
152 onChange={({ target: { value } }) =>
153 this.selectImportSource(value)
154 }
155 />
156 </div>
157 <div className={"col-4"}>
158 <Label htmlFor="importDest">Import Destination</Label>
159 <Select
160 value={this.state.selectedContentType}
161 name="importDest"
162 options={this.state.modelOptions}
163 onChange={({ target: { value } }) =>
164 this.selectImportDest(value)
165 }
166 />
167 </div>
168 </Row>
169 <Row>
170 {this.state.importSource === "upload" && (
171 <UploadFileForm
172 onRequestAnalysis={this.onRequestAnalysis}
173 loadingAnalysis={this.state.analyzing}
174 />
175 )}
176 {this.state.importSource === "url" && (
177 <ExternalUrlForm
178 onRequestAnalysis={this.onRequestAnalysis}
179 loadingAnalysis={this.state.analyzing}
180 />
181 )}
182 {this.state.importSource === "raw" && (
183 <RawInputForm
184 onRequestAnalysis={this.onRequestAnalysis}
185 loadingAnalysis={this.state.analyzing}
186 />
187 )}
188 </Row>
189 </Block>
190 </div>
191 {this.state.analysis && (
192 <Row className="row">
193 <MappingTable
194 analysis={this.state.analysis}
195 targetModel={this.getTargetModel()}
196 onChange={this.setFieldMapping}
197 />
198 </Row>
199 )}
200 </div>
201 );
202 }
203}
204export default memo(HomePage);
Congratulations! the last thing is to add an “Import Button” at the bottom of our MappingTable
component to actually “Run the Import“ for us
Button
just under your MappingTable
component tag inside HomePage/index.js
file as in below code and don't forget to import the Button
component at the top of the file:1import { Button } from "@buffetjs/core";
2
3...
4
5<Button
6 style={{ marginTop: 12 }}
7 label={"Run the Import"}
8 onClick={this.onSaveImport}
9/>
10...
onSaveImport
method as below just under your state
:1onSaveImport = async () => {
2 const { selectedContentType, fieldMapping } = this.state;
3 const { analysisConfig } = this;
4 const importConfig = {
5 ...analysisConfig,
6 contentType: selectedContentType,
7 fieldMapping
8 };
9 try {
10 await request("/import-content", { method: "POST", body: importConfig });
11 this.setState({ saving: false }, () => {
12 strapi.notification.info("Import started");
13 });
14 } catch (e) {
15 strapi.notification.error(`${e}`);
16 }
17 };
Awesome! Whenever the user clicks on “Run the Import” button, we send the “analysis result” alongside “target content type” and user provided “field mapping” to our backend API, and then our import will be initiated
Let’s setup our endpoint.
services
directory and replace ImportContent.js
file with the following:1"use strict";
2/** * ImportContent.js service
3 * * @description: A set of functions similar to controller's actions to avoid code duplication. */
4const { resolveDataFromRequest, getItemsFromData } = require("./utils/utils");
5const analyzer = require("./utils/analyzer");
6const _ = require("lodash");
7const importFields = require("./utils/importFields");
8const importMediaFiles = require("./utils/importMediaFiles");
9
10const import_queue = {};
11
12module.exports = {
13 preAnalyzeImportFile: async ctx => {
14 const { dataType, body, options } = await resolveDataFromRequest(ctx);
15 const { sourceType, items } = await getItemsFromData({
16 dataType,
17 body,
18 options
19 });
20 const analysis = analyzer.analyze(sourceType, items);
21 return { sourceType, ...analysis };
22 },
23 importItems: (importConfig, ctx) =>
24 new Promise(async (resolve, reject) => {
25 const { dataType, body } = await resolveDataFromRequest(ctx);
26 console.log("importitems", importConfig);
27 try {
28 const { items } = await getItemsFromData({
29 dataType,
30 body,
31 options: importConfig.options
32 });
33 import_queue[importConfig.id] = items;
34 } catch (error) {
35 reject(error);
36 }
37 resolve({
38 status: "import started",
39 importConfigId: importConfig.id
40 });
41 importNextItem(importConfig);
42 }),
43};
Similar to what we did in “analyzing”, we are extracting the data and parsing its items once again. Then we will queue those parsed items for further processing. What we need here is a function that iterates on that queue and stores every item.
module.exports
:1const importNextItem = async importConfig => {
2 const sourceItem = import_queue[importConfig.id].shift();
3 if (!sourceItem) {
4 console.log("import complete");
5 await strapi
6 .query("importconfig", "import-content")
7 .update({ id: importConfig.id }, { ongoing: false });
8 return;
9 }
10 try {
11 const importedItem = await importFields(
12 sourceItem,
13 importConfig.fieldMapping
14 );
15 const savedContent = await strapi
16 .query(importConfig.contentType)
17 .create(importedItem);
18 const uploadedFiles = await importMediaFiles(
19 savedContent,
20 sourceItem,
21 importConfig
22 );
23 const fileIds = _.map(_.flatten(uploadedFiles), "id");
24 await strapi.query("importeditem", "import-content").create({
25 importconfig: importConfig.id,
26 ContentId: savedContent.id,
27 ContentType: importConfig.contentType,
28 importedFiles: { fileIds }
29 });
30 } catch (e) {
31 console.log(e);
32 }
33 const { IMPORT_THROTTLE } = strapi.plugins["import-content"].config;
34 setTimeout(() => importNextItem(importConfig), IMPORT_THROTTLE);
35};
Our importing process at above code is consisted of creating the content type, importing the media files, and creating an “imported item” entry that will keep track of our created content types for the future; for instance, if we ever want to undo our imports, we must know which content types entries are related to this specific import.
Before we continue on to create our action, there are 2 functions that we are calling here: importFields
and importMediaFiles
.
Go to utils
directory and create a new file named importFields.js
:
services/utils/importFields.js
Paste the below code inside the new file:
1const striptags = require("striptags");
2const importFields = async (sourceItem, fieldMapping) => {
3 const importedItem = {};
4 Object.keys(fieldMapping).forEach(async sourceField => {
5 const { targetField, stripTags } = fieldMapping[sourceField];
6 if (!targetField || targetField === "none") {
7 return;
8 }
9 const originalValue = sourceItem[sourceField];
10 importedItem[targetField] = stripTags
11 ? striptags(originalValue)
12 : originalValue;
13 });
14 return importedItem;
15};
16module.exports = importFields;
the “field mapping” object is a relation mapping between “imported data item” fields and “target content type” fields. The above function extracts the value for the target fields and generate the candidate object for our “content type” entry on top of those fields.
Create a new file named importMediaFiles.js
in utils
directory:
services/utils/importMediaFiles.js
Paste the following:
1const _ = require("lodash");
2const request = require("request");
3const fileFromBuffer = require("./fileFromBuffer");
4const { getMediaUrlsFromFieldData } = require("../utils/fieldUtils");
5
6const fetchFiles = url =>
7 new Promise((resolve, reject) => {
8 request({ url, method: "GET", encoding: null }, async (err, res, body) => {
9 if (err) {
10 reject(err);
11 }
12 const mimeType = res.headers["content-type"].split(";").shift();
13 const parsed = new URL(url);
14 const extension = parsed.pathname
15 .split(".")
16 .pop()
17 .toLowerCase();
18 resolve(fileFromBuffer(mimeType, extension, body));
19 });
20 });
21const storeFiles = async file => {
22 const uploadProviderConfig = await strapi
23 .store({
24 environment: strapi.config.environment,
25 type: "plugin",
26 name: "upload"
27 })
28 .get({ key: "provider" });
29 return await strapi.plugins["upload"].services["upload"].upload(
30 [file],
31 uploadProviderConfig
32 );
33};
34const relateFileToContent = ({
35 contentType,
36 contentId,
37 targetField,
38 fileBuffer
39}) => {
40 fileBuffer.related = [
41 {
42 refId: contentId,
43 ref: contentType,
44 source: "content-manager",
45 field: targetField
46 }
47 ];
48 return fileBuffer;
49};
50const importMediaFiles = async (savedContent, sourceItem, importConfig) => {
51 const { fieldMapping, contentType } = importConfig;
52 const uploadedFileDescriptors = _.mapValues(
53 fieldMapping,
54 async (mapping, sourceField) => {
55 if (mapping.importMediaToField) {
56 const urls = getMediaUrlsFromFieldData(sourceItem[sourceField]);
57 const fetchPromises = _.uniq(urls).map(fetchFiles);
58 const fileBuffers = await Promise.all(fetchPromises);
59 const relatedContents = fileBuffers.map(fileBuffer =>
60 relateFileToContent({
61 contentType,
62 contentId: savedContent.id,
63 targetField: mapping.importMediaToField,
64 fileBuffer
65 })
66 );
67 const storePromises = relatedContents.map(storeFiles);
68 const storedFiles = await Promise.all(storePromises);
69 console.log(_.flatten(storedFiles));
70 return storedFiles;
71 }
72 }
73 );
74 return await Promise.all(_.values(uploadedFileDescriptors));
75};
76module.exports = importMediaFiles;
What we are doing here is to check if we have importMediaToField
specified on the mapping
or not, and in case we have, we first extract the urls
, then fetch the files for those urls
, relate them to the target content type and store them as well.
Create a new file named fileFromBuffer.js
in utils
directory:
services/utils/fileFromBuffer.js
Paste the following code inside that file:
1const crypto = require('crypto')
2const uuid = require("uuid/v4");
3
4function niceHash(buffer) {
5 return crypto
6 .createHash("sha256")
7 .update(buffer)
8 .digest("base64")
9 .replace(/=/g, "")
10 .replace(/\//g, "-")
11 .replace(/\+/, "_");
12}
13const fileFromBuffer = (mimeType, extension, buffer) => {
14 const fid = uuid();
15 return {
16 buffer,
17 sha256: niceHash(buffer),
18 hash: fid.replace(/-/g, ""),
19 name: `${fid}.${extension}`,
20 ext: `.${extension}`,
21 mime: mimeType,
22 size: (buffer.length / 1000).toFixed(2)
23 };
24};
25module.exports = fileFromBuffer;
our service is ready!
ImportContent.js
file inside controllers
directory and add the following function:1create: async ctx => {
2 const services = strapi.plugins["import-content"].services;
3 const importConfig = ctx.request.body;
4 importConfig.ongoing = true;
5 const record = await strapi
6 .query("importconfig", "import-content")
7 .create(importConfig);
8 console.log("create", record);
9 await services["importcontent"].importItems(record, ctx);
10 ctx.send(record);
11}
For each import, we are storing a specific config entry called “import config”. This will help us keep details of our import like the date of this import and the identification of all the content types entries that are created because of this import.
routes.json
inside config
directory and define our new route:1{
2 "method": "POST",
3 "path": "/",
4 "handler": "ImportContent.create",
5 "config": {
6 "policies": []
7 }
8}
Awesome! Our import endpoint is implemented; The main purpose of this plugin is accomplished. The rest is to show our initiated imports to the users. Also we are going to allow our users to “undo” the initiated imports as well as “removing” their related configs.
ImportContent.js
file inside controllers
directory:1index: async ctx => {
2 const entries = await strapi.query("importconfig", "import-content").find();
3 const withCounts = entries.map(entry => ({
4 ...entry,
5 importedCount: entry.importeditems.length,
6 importeditems: []
7 }));
8 const withName = withCounts.map(entry =>
9 ({
10 ...entry,
11 contentType: strapi.contentTypes[entry.contentType].info.name ||
12 entry.contentType
13 }))
14 ctx.send(withName);
15}
it will retrieve the stored configs for user initiated imports. This will allow us to provide our users with a history of their imports and show some details about each import. Beside retrieving the import configs, it would be nice to have a way to delete them as well.
1 delete: async ctx => {
2 const importId = ctx.params.importId;
3 const res = await strapi.query("importconfig", "import-content").delete({
4 id: importId
5 });
6 if (res && res.id) {
7 ctx.send(res.id);
8 } else {
9 ctx.response.status = 400;
10 ctx.response.message = "could not delete: the provided id might be wrong";
11 }
12 }
The only thing left is a way to undo our imports. For this we are about to implement a service similar to what we did for importing data.
ImportContent.js
file in services
directory, and add the following before the module.exports
:1const undo_queue = {};
2const removeImportedFiles = async (fileIds, uploadConfig) => {
3 const removePromises = fileIds.map(id =>
4 strapi.plugins["upload"].services.upload.remove({ id }, uploadConfig)
5 );
6 return await Promise.all(removePromises);
7};
8const undoNextItem = async (importConfig, uploadConfig) => {
9 const item = undo_queue[importConfig.id].shift();
10 if (!item) {
11 console.log("undo complete");
12 await strapi
13 .query("importconfig", "import-content")
14 .update({ id: importConfig.id }, { ongoing: false });
15 return;
16 }
17 try {
18 await strapi.query(importConfig.contentType).delete({ id: item.ContentId });
19 } catch (e) {
20 console.log(e);
21 }
22 try {
23 const importedFileIds = _.compact(item.importedFiles.fileIds);
24 await removeImportedFiles(importedFileIds, uploadConfig);
25 } catch (e) {
26 console.log(e);
27 }
28 try {
29 await strapi.query("importeditem", "import-content").delete({
30 id: item.id
31 });
32 } catch (e) {
33 console.log(e);
34 }
35 const { UNDO_THROTTLE } = strapi.plugins["import-content"].config;
36 setTimeout(() => undoNextItem(importConfig, uploadConfig), UNDO_THROTTLE);
37};
The undoNextItem
function is iterating on the queue for removing every imported item. This undo process is consisted of removing the content type entry, removing the related files and the imported item entry as well.
module.exports
:1undoItems: importConfig =>
2 new Promise(async (resolve, reject) => {
3 try {
4 undo_queue[importConfig.id] = importConfig.importeditems;
5 } catch (error) {
6 reject(error);
7 }
8 await strapi
9 .query("importconfig", "import-content")
10 .update({ id: importConfig.id }, { ongoing: true });
11 resolve({
12 status: "undo started",
13 importConfigId: importConfig.id
14 });
15 const uploadConfig = await strapi
16 .store({
17 environment: strapi.config.environment,
18 type: "plugin",
19 name: "upload"
20 })
21 .get({ key: "provider" });
22 undoNextItem(importConfig, uploadConfig);
23}),
well done!
ImportContent.js
file inside controllers
directory and add the following function:1undo: async ctx => {
2 const services = strapi.plugins["import-content"].services;
3 const importId = ctx.params.importId;
4 const importConfig = await strapi
5 .query("importconfig", "import-content")
6 .findOne({ id: importId });
7 console.log("undo", importId);
8 await services["importcontent"].undoItems(importConfig);
9 ctx.send(importConfig);
10}
routes.json
file at config
directory and append the following routes:1 {
2 "method": "GET",
3 "path": "/",
4 "handler": "ImportContent.index",
5 "config": {
6 "policies": []
7 }
8 },
9 {
10 "method": "DELETE",
11 "path": "/:importId",
12 "handler": "ImportContent.delete",
13 "config": {
14 "policies": []
15 }
16 },
17 {
18 "method": "POST",
19 "path": "/:importId/undo",
20 "handler": "ImportContent.undo",
21 "config": {
22 "policies": []
23 }
24 }
Now that we have all the endpoints we need, let’s go back to our admin
directory and implement the HistoryPage
. This page will show a history of all the user initiated imports, their status and 2 buttons for undo
and delete
actions.
Warning! while we are OK to continue to the next section, there is one point we have to mention here: Until now we have only provided endpoints for the functionalities that we already have like "analyzing", "importing" and running an "undo" operation. The point is that if we want to be consistent with other plugins like "graphql", we must define a set of controllers for each one of our content types which in our case are importconfig
& importeditem
. These controllers are the famous set of find
, findOne
, count
, update
and delete
which i bet you are already familiar with them. So in case that you want this plugin to be efficient in production environments, you must make it consistent. Before jumping to the next section, please do the followings:
ImportConfig
in controllers
directory:controllers/ImportConfig.js
1module.exports = {
2 count: async ctx => {
3 const entries = await strapi.query("importconfig", "import-content").count(ctx.request.query)
4 ctx.send(entries)
5 },
6 findOne: async ctx => {
7 const entries = await strapi.query("importconfig", "import-content").findOne({id: ctx.params.importId})
8 ctx.send(entries)
9 },
10 find: async ctx => {
11 const entries = await strapi.query("importconfig", "import-content").find(ctx.request.query)
12 ctx.send(entries)
13 },
14 delete: async ctx => {
15 return strapi.query("importconfig", "import-content").delete({id: ctx.params.importId})
16 },
17 update: async ctx => {
18 return strapi.query("importconfig", "import-content").update({id: ctx.params.importId}, ctx.request.body)
19 }
20};
ImportedItem
in controllers
directory:controllers/ImportedItem.js
1module.exports = {
2 count: async ctx => {
3 const entries = await strapi.query("importeditem", "import-content").count(ctx.request.query)
4 ctx.send(entries)
5 },
6 findOne: async ctx => {
7 const entries = await strapi.query("importeditem", "import-content").findOne({id: ctx.params.importId})
8 ctx.send(entries)
9 },
10 find: async ctx => {
11 const entries = await strapi.query("importeditem", "import-content").find(ctx.request.query)
12 ctx.send(entries)
13 },
14 delete: async ctx => {
15 return strapi.query("importeditem", "import-content").delete({id: ctx.params.importId})
16 },
17 update: async ctx => {
18 return strapi.query("importeditem", "import-content").update({id: ctx.params.importId}, ctx.request.body)
19 }
20};
routes.json
:config/routes.json
1{
2 "method": "GET",
3 "path": "/importconfig",
4 "handler": "ImportConfig.find",
5 "config": {
6 "policies": []
7 }
8},
9{
10 "method": "GET",
11 "path": "/importconfig/:importId",
12 "handler": "ImportConfig.findOne",
13 "config": {
14 "policies": []
15 }
16},
17{
18 "method": "GET",
19 "path": "/importconfig/count",
20 "handler": "ImportConfig.count",
21 "config": {
22 "policies": []
23 }
24},
25{
26 "method": "PUT",
27 "path": "/importconfig/:importId",
28 "handler": "ImportConfig.update",
29 "config": {
30 "policies": []
31 }
32},
33{
34 "method": "DELETE",
35 "path": "/importconfig/:importId",
36 "handler": "ImportConfig.delete",
37 "config": {
38 "policies": []
39 }
40},
41{
42 "method": "GET",
43 "path": "/importeditem",
44 "handler": "ImportedItem.find",
45 "config": {
46 "policies": []
47 }
48},
49{
50 "method": "GET",
51 "path": "/importeditem/:importId",
52 "handler": "ImportedItem.findOne",
53 "config": {
54 "policies": []
55 }
56},
57{
58 "method": "GET",
59 "path": "/importeditem/count",
60 "handler": "ImportedItem.count",
61 "config": {
62 "policies": []
63 }
64},
65{
66 "method": "PUT",
67 "path": "/importeditem/:importId",
68 "handler": "ImportedItem.update",
69 "config": {
70 "policies": []
71 }
72},
73{
74 "method": "DELETE",
75 "path": "/importeditem/:importId",
76 "handler": "ImportedItem.delete",
77 "config": {
78 "policies": []
79 }
80}
Great you finished the third part of this tutorial! Let's create the History page! https://strapi.io/blog/how-to-create-an-import-content-plugin-part-4-4
Pouya is an active member of the Strapi community, who has been contributing actively with contributions in the core and plugins.