Working with imports¶
Importing data into Elevate from a retailer's systems is a key action that has two conceptual parts, an import method and an import scheduling. The import method defines how an import should be managed and the import scheduling defines when imports should be performed.
These parts both impact system performance in different ways. The level of complexity of the implementation also differ based on both method and scheduling. It is the responsibility of the retailer and/or their implementing partner to enable transfer and modification of data from the retailer's systems into Voyado Elevate 3. Imports are performed on half of a retailer's cluster/s at a time to avoid missing information on the retailer's site/s.
Imports are, apart from synonyms for the Web API v1, performed by posting XML via the RESTful API import method. All XML must be encoded using UTF-8 and include the operations that is to be performed and all other data type related information.
Compression¶
It is recommended that imports are compressed with gzip.
Operations¶
An import XML have four different operations that Elevate can act on. The operations, and their order of use, are clear, remove, add, and update. These operations are used in different combinations in the import methods.
Add¶
The add operation includes a list of attributes and values for the data entity in the XML. Each data entity in the operation must have its mandatory key attribute included, such as product_key for a product.
If a data entity in an add operation have a key that already exist in Elevate, the entity in the import will overwrite the existing one. The variants of the old product will remain and now be attached to the new product.
To overwrite a product and remove the current attached variants, it is recommended to use the remove operation on the entire product before re-adding the data to Elevate.
Add operation example
<?xml version='1.0' encoding='UTF-8'?>
<operations>
<add>
<product>
<product_key>1234568-1</product_key>
<title>NikeCourt Air Zoom Ultra Rct</title>
<url>/products/1234568</url>
<has_image>true</has_image>
<image_url>/images/products/1234568-1.jpg</image_url>
<price>109.00</price>
<market>UK</market>
<locale>en-GB</locale>
<release_date>2018-03-10</release_date>
<section>men-shoes-tennis-shoes</section>
<brand>Nike</brand>
<color>White/Grey/Blue</color>
<variants>
<variant>
<variant_key>1234568-1-42</variant_key>
<size>42</size>
<in_stock>true</in_stock>
</variant>
<variant>
<variant_key>1234568-1-43</variant_key>
<size>43</size>
<in_stock>true</in_stock>
</variant>
</variants>
</product>
</add>
</operations>
Update¶
The update operation allows for modification of attribute values for an existing product, variant, or category. The key attribute, such as product_key, is used to identify the entity that is to be updated. Note that the update operation can not be performed with ad entities.
The add operation should always be used before an update operation if both operations are present in the same import file. Entities in an updateoperation in the import file without key attributes present in Elevate will not be added to Elevate automatically.
New attributes will be set as unconfigured if added to an existing entity using the update operation.
The following example updates the value of the price attribute for the product and the value of the in_stock attribute for the variant.
Update operation example
<?xml version='1.0' encoding='UTF-8'?>
<operations>
<update>
<product>
<product_key>1234567-1</product_key>
<price>89.00</price>
</product>
<variant>
<variant_key>1234567-1-43</variant_key>
<in_stock>false</in_stock>
</variant>
</update>
</operations>
Remove¶
The remove operation removes existing data from Elevate. The key attribute, such as product_key, is used to identify the entity that is to be removed. The remove operation should be performed before both the add and update operations.
If a product is removed, its connected variants will also be removed from Elevate. A category cannot be removed if any product is linked to it, or if it is the parent to another category. It can however be removed and added again in the same import. It is also possible to remove an entire tree as long as the categories in the tree are not linked to any products.
The following example removes a product, a variant, an ad, and a category from Elevate.
Remove operation example
<?xml version='1.0' encoding='UTF-8'?>
<operations>
<remove>
<product>
<product_key>1234567-2</product_key>
</product>
<variant>
<variant_key>1234567-1-43</variant_key>
</variant>
<ad>
<ad_key>ad2018030120180401</ad_key>
</ad>
<!-- Removes a category in section tree -->
<category_tree product_attribute="section">
<category key="men-shoes-golf-shoes" />
</category_tree>
<!-- Removes an entire section tree -->
<category_tree product_attribute="section"/>
</remove>
</operations>
Clear¶
The clear operation is used to clear a data type before an add operation follows. A clear operation can not be done after any other operation. The clear operation removes all data related to the data type in the operation and does not accept any key attributes. The following example show how clear is used to remove all product and category_tree data.
Clear operation example
<?xml version='1.0' encoding='UTF-8'?>
<operations>
<clear>
<product/>
<category_tree/>
</clear>
</operations>
Filters can also be used with the clear operation to remove multiple products and variants without specifying each entity. The operation will then remove all products and variants that match the filter. To remove specific entities using key attributes, see Remove.
Clear operation with filter example
<?xml version='1.0' encoding='UTF-8'?>
<operations>
<clear>
<product>
<filter>section:'men-shoes-tennis-shoes' AND in_stock:'false'</filter>
</product>
</clear>
</operations>
Import methods¶
Full import¶
A full import uses the clear and add operations and is the import method that can be considered the easiest to implement. The import method does not affect behavior data, i.e. no behavior data is removed when using clear. A clear operation to remove all products must be performed at the start of a full import, see the Full import example below.
When performing a full import there will be delay before product changes are visible on a site. The delay and import time is also dependant on the size of the product catalog.
Verify operations in imports
To avoid missing products and site downtime, verify that there is an add operation with the necessary data after the clear operation.
Full import example
<?xml version='1.0' encoding='UTF-8'?>
<operations>
<clear>
<product/>
</clear>
<add>
<product>
<product_key>1234567-1</product_key>
<title>Free Run 2</title>
<url>/products/1234567</url>
<has_image>true</has_image>
<image_url>/images/products/1234567-1.jpg</image_url>
<price>99.00</price>
<market>UK</market>
<locale>en-GB</locale>
<release_date>2017-03-10</release_date>
<section>men-shoes-running-shoes</section>
<brand>Nike</brand>
<color>Black</color>
<variants>
<variant>
<variant_key>1234567-1-42</variant_key>
<size>42</size>
<in_stock>true</in_stock>
</variant>
<variant>
<variant_key>1234567-1-43</variant_key>
<size>43</size>
<in_stock>true</in_stock>
</variant>
</variants>
</product>
</add>
</operations>
Partial add import¶
The partial add import use the add operation to completely overwrite data based on the key attribute, such as product_key. Existing attributes in Elevate that is not present in the import will be removed. The partial add import has better performance than the full import.
Entities can be removed during a partial add import if desired. If so, the remove operation is to be used before the add operation. The key attribute, such as product_key, is used to identify the entity that is to be removed. If a product is removed, its connected variants will also be removed from Elevate. To overwrite a product and remove the current attached variants, it is recommended to use the remove operation on the entire product before re-adding the data to Elevate.
Partial add import with remove example
<?xml version='1.0' encoding='UTF-8'?>
<operations>
<remove>
<product>
<product_key>12345688-1</product_key>
</product>
<product>
<product_key>12345695-3</product_key>
</product>
<product>
<product_key>12345701-1</product_key>
</product>
</remove>
<add>
<product>
<product_key>1234569-1</product_key>
<title>Nike Air Zoom Vapor X Clay</title>
<url>/products/1234569</url>
<has_image>true</has_image>
<image_url>/images/products/1234569-1.jpg</image_url>
<price>140.00</price>
<market>UK</market>
<locale>en-GB</locale>
<release_date>2018-03-10</release_date>
<section>men-shoes-tennis-shoes</section>
<brand>Nike</brand>
<color>Pink/Black</color>
<variants>
<variant>
<variant_key>1234569-1-42</variant_key>
<size>42</size>
<in_stock>true</in_stock>
</variant>
<variant>
<variant_key>1234569-1-43</variant_key>
<size>43</size>
<in_stock>true</in_stock>
</variant>
</variants>
</product>
</add>
</operations>
Partial update import¶
The partial update import use the update operation to only update the data that has changed. The key attribute, such as product_key, is used to identify the entity that is to be updated or removed. Existing attributes in Elevate that is not present in the import will remain unchanged.
The partial update import offers the best performance but is also the most complex import method to implement. The retailer's systems must track changes on an attribute level to only update what has changed.
Entities can be removed during a partial update import if desired. If so, the remove operation is to be used before the update operation. The key attribute, such as product_key, is used to identify the entity that is to be removed. If a product is removed, its connected variants will also be removed from Elevate.
The add operation can also be included in a partial update import if necessary. It should then be performed before the update operation and if applicable, after the remove operation.
Partial update import with remove
<?xml version='1.0' encoding='UTF-8'?>
<operations>
<remove>
<product>
<product_key>12345678-2</product_key>
</product>
<variant>
<variant_key>1234567-1-44</variant_key>
<product_key>1234567-1</product_key>
</variant>
</remove>
<update>
<product>
<product_key>1234567-1</product_key>
<price>79.00</price>
</product>
<variant>
<variant_key>1234567-1-42</variant_key>
<product_key>1234567-1</product_key>
<in_stock>false</in_stock>
</variant>
</update>
</operations>
Import scheduling¶
Imports can be performed dynamically based on events or at fixed intervals. Only one import can be processed by a cluster at any given time.
Interval based import¶
Interval based imports are performed at fixed times and/or intervals. All changes during an interval are combined in one import file and posted via the RESTful API.
When performing a interval based imports there will be delay before product changes are visible on a site. The delay and import time is also dependant on the size of the import file.
Event based import¶
Event based imports are triggered when certain predefined events happen on a retailers site. Compared to interval based imports, changes to a site will reflect much faster. As only one import can be processed by a cluster at any given times, overlapping changes must be queued and optimally batched.
The data model must be optimized to avoid too many updates as that affects overall performance. An example of this is using a stock status with high, low, sold out, instead of actual available items.
Import implementation¶
Data exported from a retailer's systems may require modification to be compatible with Voyado Elevate. Implementation and management of functions for export data modification, and data imports into Elevate, is the responsibility of the retailer or their implementing partner.
Data entity types¶
Products¶
All product entities must have a unique product_key attribute. All variants must have a unique variant_key attribute, and if a variant is a stand-alone element, it must also include a product_key attribute.
If a product is imported and a product using the same product_key value already exist in Elevate, the existing product will be overwritten by the import. Variants connected to the old product will be connected to the new product.
Full import example (nested variants)
<?xml version='1.0' encoding='UTF-8'?>
<operations>
<clear>
<product/>
</clear>
<add>
<product>
<product_key>1234567-1</product_key>
<title>Free Run 2</title>
<url>/products/1234567</url>
<has_image>true</has_image>
<image_url>/images/products/1234567-1.jpg</image_url>
<price>99.00</price>
<market>UK</market>
<locale>en-GB</locale>
<release_date>2017-03-10</release_date>
<section>men-shoes-running-shoes</section>
<brand>Nike</brand>
<color>Black</color>
<variants>
<variant>
<variant_key>1234567-1-42</variant_key>
<size>42</size>
<in_stock>true</in_stock>
</variant>
<variant>
<variant_key>1234567-1-43</variant_key>
<size>43</size>
<in_stock>true</in_stock>
</variant>
</variants>
</product>
<product>
<product_key>1234567-2</product_key>
<title>Free Run 2</title>
<url>/products/1234567/2</url>
<has_image>true</has_image>
<image_url>/images/products/1234567-2.jpg</image_url>
<price>99.00</price>
<market>UK</market>
<locale>en-GB</locale>
<release_date>2017-03-10</release_date>
<section>men-shoes-running-shoes</section>
<brand>Nike</brand>
<color>White</color>
<variants>
<variant>
<variant_key>1234567-2-42</variant_key>
<size>42</size>
<in_stock>false</in_stock>
</variant>
<variant>
<variant_key>1234567-2-43</variant_key>
<size>43</size>
<in_stock>true</in_stock>
</variant>
</variants>
</product>
</add>
</operations>
Full import example (stand-alone variants)
<?xml version='1.0' encoding='UTF-8'?>
<operations>
<clear>
<product/>
</clear>
<add>
<product>
<product_key>1234567-1</product_key>
<title>Free Run 2</title>
<url>/products/1234567</url>
<has_image>true</has_image>
<image_url>/images/products/1234567-1.jpg</image_url>
<price>99.00</price>
<market>UK</market>
<locale>en-GB</locale>
<release_date>2017-03-10</release_date>
<section>men-shoes-running-shoes</section>
<brand>Nike</brand>
<color>Black</color>
</product>
<product>
<product_key>1234567-2</product_key>
<title>Free Run 2</title>
<url>/products/1234567/2</url>
<has_image>true</has_image>
<image_url>/images/products/1234567-2.jpg</image_url>
<price>99.00</price>
<market>UK</market>
<locale>en-GB</locale>
<release_date>2017-03-10</release_date>
<section>men-shoes-running-shoes</section>
<brand>Nike</brand>
<color>White</color>
</product>
<variant>
<variant_key>1234567-1-42</variant_key>
<product_key>1234567-1</product_key>
<size>42</size>
<in_stock>true</in_stock>
</variant>
<variant>
<variant_key>1234567-1-43</variant_key>
<product_key>1234567-1</product_key>
<size>43</size>
<in_stock>true</in_stock>
</variant>
<variant>
<variant_key>1234567-2-42</variant_key>
<product_key>1234567-2</product_key>
<size>42</size>
<in_stock>false</in_stock>
</variant>
<variant>
<variant_key>1234567-2-43</variant_key>
<product_key>1234567-2</product_key>
<size>43</size>
<in_stock>true</in_stock>
</variant>
</add>
</operations>
Partial add import example
<?xml version='1.0' encoding='UTF-8'?>
<operations>
<remove>
<product>
<product_key>1234567-1</product_key>
</product>
<product>
<product_key>1234567-2</product_key>
</product>
</remove>
<add>
<product>
<product_key>1234569-1</product_key>
<title>Nike Air Zoom Vapor X Clay</title>
<url>/products/1234569</url>
<has_image>true</has_image>
<image_url>/images/products/1234569-1.jpg</image_url>
<price>140.00</price>
<market>UK</market>
<locale>en-GB</locale>
<release_date>2018-03-10</release_date>
<section>men-shoes-tennis-shoes</section>
<brand>Nike</brand>
<color>Pink/Black</color>
<variants>
<variant>
<variant_key>1234569-1-42</variant_key>
<size>42</size>
<in_stock>true</in_stock>
</variant>
<variant>
<variant_key>1234569-1-43</variant_key>
<size>43</size>
<in_stock>true</in_stock>
</variant>
</variants>
</product>
</add>
</operations>
Partial update import example
<?xml version='1.0' encoding='UTF-8'?>
<operations>
<remove>
<product>
<product_key>1234567-1</product_key>
</product>
<product>
<product_key>1234567-2</product_key>
</product>
</remove>
<update>
<product>
<product_key>1234569-1</product_key>
<price>130.00</price>
</product>
<variant>
<variant_key>1234569-1-43</variant_key>
<in_stock>false</in_stock>
</variant>
</update>
</operations>
Categories¶
All category entities must have a key and a parent attribute. The key must be unique within its category tree.
A category cannot be removed if any product is linked to it, or if it is the parent to another category. It can however be removed and added again in the same import. It is also possible to remove an entire tree as long as the categories in the tree are not linked to any products.
Full import example
<?xml version='1.0' encoding='UTF-8'?>
<operations>
<clear>
<category_tree/>
</clear>
<add>
<category_tree product_attribute="section">
<category key="men-shoes" parent="root">
<display_name>Mens shoes</display_name>
<locale>en-GB</locale>
<attributes>
<market>UK</market>
<url>/categories/mens-shoes</url>
<department>Shoes</department>
</attributes>
</category>
<category key="men-shoes-running-shoes" parent="men-shoes">
<display_name>Running shoes</display_name>
<locale>en-GB</locale>
<attributes>
<market>UK</market>
<url>/categories/men-shoes-running-shoes</url>
<department>Men</department>
</attributes>
</category>
<category key="men-shoes-tennis-shoes" parent="men-shoes">
<display_name>Tennis shoes</display_name>
<locale>en-GB</locale>
<attributes>
<market>UK</market>
<url>/categories/men-shoes-tennis-shoes</url>
<department>Men</department>
</attributes>
</category>
</category_tree>
</add>
</operations>
Partial add import example
<?xml version='1.0' encoding='UTF-8'?>
<operations>
<remove>
<category_tree product_attribute="section">
<category key="men-shoes-golf-shoes" />
<category key="men-shoes-hiking-shoes" />
</category_tree>
</remove>
<add>
<category_tree product_attribute="section">
<category key="men-shoes-basketball-shoes" parent="men-shoes">
<display_name>Basketball shoes</display_name>
<locale>en-GB</locale>
<attributes>
<market>UK</market>
<url>/categories/men-shoes-basketball-shoes</url>
<department>Men</department>
</attributes>
</category>
<category key="men-shoes-lifestyle-shoes" parent="men-shoes">
<display_name>Sneakers</display_name>
<locale>en-GB</locale>
<attributes>
<market>UK</market>
<url>/categories/men-shoes-sneakers</url>
<department>Men</department>
</attributes>
</category>
</category_tree>
</add>
</operations>
Partial update import example
<?xml version='1.0' encoding='UTF-8'?>
<operations>
<remove>
<category_tree product_attribute="section">
<category key="men-shoes-golf-shoes" />
<category key="men-shoes-hiking-shoes" />
</category_tree>
<category_tree product_attribute="sandals" />
</remove>
<update>
<category_tree product_attribute="section">
<category key="men-shoes-lifestyle-shoes" parent="men-shoes">
<display_name>Lifestyle shoes</display_name>
<attributes>
<url>/categories/men-shoes-lifestyle-shoes/</url>
</attributes>
</category>
</category_tree>
</update>
</operations>
Ads¶
All ad entities must have a unique ad_key attribute. The following examples include standard and common additional attributes used with ads. Note that the update operation can not be performed with ad entities.
Full import example
<?xml version='1.0' encoding='UTF-8'?>
<operations>
<clear>
<ad/>
</clear>
<add>
<ad>
<ad_key>ad2018011420180214</ad_key>
<start_time>2018-01-14T08:00:00+01:00</start_time>
<end_time>2018-02-15T24:00:00+01:00</end_time>
<included>category:'cd' AND genre:'soul' AND genre:'rnb'</included>
<related>category:'cd' AND genre:'jazz'</related>
<title>Valentines day music offers</title>
<market>UK</market>
<locale>en-GB</locale>
<tags>cd,music</tags>
<live_products>3</live_products>
<campaign_key>valentines2018</campaign_key>
<campaign_url>/campaigns/valentines-2018/music/</campaign_url>
<format>fullwidth</format>
<image_url>/images/ads/ad2018011420180214.jpg</image_url>
<ad_category>frontpage</ad_category>
</ad>
<ad>
<ad_key>ad2018062520180625</ad_key>
<start_time>2018-06-25T08:00:00+01:00</start_time>
<end_time>2018-07-25T24:00:00+01:00</end_time>
<included>category:'cd'</included>
<related>UNIVERSE</related>
<title>Summer CD Sale</title>
<market>UK</market>
<locale>en-GB</locale>
<tags>cd,music</tags>
<live_products>3</live_products>
<campaign_key>summersalemusic2018</campaign_key>
<campaign_url>/campaigns/summer-2018/music/</campaign_url>
<format>fullheight</format>
<image_url>/images/ads/ad2018062520180625.jpg</image_url>
<ad_category>frontpage</ad_category>
</ad>
</add>
</operations>
Partial add import example
<?xml version='1.0' encoding='UTF-8'?>
<operations>
<remove>
<ad><ad_key>ad2018011420180214</ad_key></ad>
</remove>
<add>
<ad>
<ad_key>ad2018062520180625</ad_key>
<start_time>2018-06-25T08:00:00+01:00</start_time>
<end_time>2018-07-25T24:00:00+01:00</end_time>
<included>category:'cd'</included>
<related>UNIVERSE</related>
<title>Summer CD Sale</title>
<market>UK</market>
<locale>en-GB</locale>
<tags>cd,music</tags>
<live_products>3</live_products>
<campaign_key>summersalemusic2018</campaign_key>
<campaign_url>/campaigns/summer-2018/music/</campaign_url>
<format>fullheight</format>
<image_url>/images/ads/ad2018062520180625.jpg</image_url>
<ad_category>frontpage</ad_category>
</ad>
<ad>
<ad_key>ad2018030120180401</ad_key>
<start_time>2018-03-01T08:00:00+01:00</start_time>
<end_time>2018-04-01T24:00:00+01:00</end_time>
<included>category:'cd' AND genre:'metal'</included>
<related>UNIVERSE</related>
<title>Metal March Sale</title>
<market>UK</market>
<locale>en-GB</locale>
<tags>cd,music</tags>
<live_products>3</live_products>
<campaign_key>metalmarchmusic2018</campaign_key>
<campaign_url>/campaigns/metal-march-2018/</campaign_url>
<format>fullwidth</format>
<image_url>/images/ads/ad2018030120180401.jpg</image_url>
<ad_category>sidebars</ad_category>
</ad>
</add>
</operations>
Synonyms¶
Synonyms can be imported into Elevate in different ways depending on what version of the Web API is used. For users of the Web API v2, imports of synonyms can be performed either via the Experience app or the RESTful API. Users of the Web API v1 are recommended to use the Experience app as the data structure of synonyms differ between Web API v1 and v2.
Special characters
Special characters in synonyms must be escaped using a preceding backslash, for example \' and \".
Import via the Experience app¶
Synonyms are imported in the Experience app by uploading an Excel-file formatted as the example below. Every line is a new synonym entry and each line must have a valid locale, a search phrase, and one or more synonyms.
Column A is the locale of the synonym, such as en-GB. If the synonym is valid for all locales, the value global should be used.
Column B is the search phrase the synonym is for.
Column C and onwards are the synonyms to be used for the search phrase.
A | B | C | D | E |
----------------------------------------------------------------
en-GB | Search phrase | Synonym | Synonym | Synonym | ...
sv-SE | Search phrase | Synonym | Synonym | Synonym | ...
global | E.g. names, brands | Synonym | Synonym | Synonym | ...
...
If a search phrase is already present for a locale in Elevate, the import will add the new synonyms to the list of synonyms for that search phrase. No search phrase or synonym information is overwritten, edited, or removed in an import via the Experience app.
Editing and removal must be done per search phrase and locale when using the Experience app.
Import via the Web API v2¶
All synonym entities must have locale, a search_phrase, and a synonym_phrase when added. Synonyms that are removed are identified by both the locale and the search_phrase. A synonym can be disabled, but not removed, by adding disabled to the entity in the import.
Full import example (Web API v2 only)
<?xml version='1.0' encoding='UTF-8'?>
<operations>
<clear>
<synonym/>
</clear>
<add>
<synonym>
<locale>en-GB</locale>
<search_phrase>Shoes</search_phrase>
<synonym_phrase>Boots</synonym_phrase>
</synonym>
<synonym>
<locale>en-GB</locale>
<search_phrase>Shoes</search_phrase>
<synonym_phrase>Sneakers</synonym_phrase>
</synonym>
</add>
</operations>
Partial add import example with disabled synonym (Web API v2 only)
<?xml version='1.0' encoding='UTF-8'?>
<operations>
<remove>
<synonym>
<locale>en-GB</locale>
<search_phrase>Shoes</search_phrase>
</synonym>
</remove>
<add>
<synonym>
<locale>en-GB</locale>
<search_phrase>Skirt</search_phrase>
<synonym_phrase>Dress</synonym_phrase>
</synonym>
<synonym>
<locale>en-GB</locale>
<search_phrase>Skirt</search_phrase>
<synonym_phrase>Kilt</synonym_phrase>
<disabled/>
</synonym>
</add>
</operations>
Configuration¶
Importing a configuration file to an Elevate cluster is recommended to be made by way of publishing it with the Elevate Manager. The configuration file is itself generated by the Elevate Manager and should be left unchanged and not edited outside of the Elevate Manager.
When a configuration file is published or imported it will overwrite the current active configuration.
Error handling¶
Force imports¶
Protection from potentially harmful product imports is built into Elevate 3 Enterprise. If an import looks dangerous, i.e. it reduces the product count to less than half, it will be rejected with an appropriate error message and code 400. To override the protection and force Elevate to accept such an import, a force=true argument can be added to the post request of the import.
#!/bin/bash
curl -i \
-X POST \
-T products.xml \
-H "Content-Type: application/xml" \
-H "Api-Key: {PRIVATE-KEY}" \
"https://{cluster-id}.api.esales.apptus.cloud/api/v2/import/products?name=prod-import&force=true"