You are here: Integrating With CPQ Cloud > Integration Guides > Bulk Uploads

Bulk Uploads

OVERVIEW

Bulk Upload is used for several reasons.

There are several ways to move data and files in CPQ Cloud. See the table below to determine which help topic to open for more information.

To Move... See...
Data to CPQ Cloud tables, in bulk This document
Data from CPQ Cloud tables, in bulk Bulk Downloads
Data Tables Import & Export of Data Tables
File Manager files, such as images, .PDFs, or XSL files File Manager
Documents in Document Engine Document Migration: Document Engine
Migration changes between CPQ Cloud environments Migration Management Overview

General Requirements

ClosedData Types Available for Bulk Upload and Download

The following table shows the Data Types that can be uploaded and downloaded, and in which format.

All data that can be uploaded into the system can also be downloaded. For the XML data type, all downloaded data can be uploaded. However, only some downloaded CSV data can be uploaded.

Data Type XML CSV
Accounts X X
Company X X
Configurable Attributes X X
File Manager* X X
Part Associations X X
Part Extended Descriptions X X
Parts X X
Price Book Associations X X
Serial Number Associations X X
Serial Numbers X X
User** X X
Part Pricing Rule Information   X
Translations   X
Account/User Associations X  
Commerce Processes*** X  
Commerce Transactions**** X  
Configuration Rules X  
Data Tables* X  
Library Functions X  
Printer Friendly Document X  
Product Definition X  
Text Administration X  

* Downloading Data Tables and File Manager is useful if you want to download all Data Tables or all File Manager files in a single ZIP file. However, specific tools, besides Bulk Upload and Download, are used to manage Data Tables and File Manager. See Import & Export of Data Tables and File Manager for more information.

** Users cannot be deleted. Instead, change a user's status to Inactive..

*** The Commerce Process can also be obtained from the Process Definition Deployment center, by using "Import Process".

**** Commerce Transactions cannot be added or deleted, so the <action> tag can only be update, and the <action_var_name> tag must be populated with this valid action.


ClosedTransferring the Product Family Data Type

The following data types require that a Product Family be specified in order to be transferred between sites, such as through migration or bulk download/upload:

Data Type XML CSV
Model (requires Product Family) X X
Product Line (requires Product Family) X X
Catalog Data (requires Product Family) X  

Configuration Data (requires Product Family)

X  
  1. Migrate the Product Definition first, because it contains the information of what Product Family has been defined.

    This will add the reference of the Product Family to the site.

    If the Product Family already exists on the site, then you can see the categories that are listed.

  2. Manually add the Product Family under Catalog Definition.
  3. Continue migrating the Product Line.
  4. Migrate the Catalog.
  5. Migrate Configuration data.
  6. Migrate Attributes.
  7. Migrate Rules.

ClosedBulk Uploads & Natural Keys

Data with the same Natural Keys as pre-existing data in the table is considered a duplicate record. For more information, see Natural Keys and Data Table Imports.


ADMINISTRATION

Creating Bulk Upload Files

The easiest way to create a file for bulk upload is to download a file for the same data type, and then use that downloaded file as a template. Just add your data to the template to create the upload file.

ClosedCreating CSV Bulk Upload Files

  1. Using a downloaded file as a template, create a CSV file containing the records you want to change.

    We recommend that you create a different file for each data type.

      All values, including leading and trailing spaces, are trimmed by the system.

      The default value for all fields with a specified value is an empty string.
      Upload of Multi-byte characters is not supported for CSV.
  2. In the Action column, for each row of data, enter an Action type from the table below.

    Action Requirements Example
    add All tags downloaded can be used to add new data. Example: <action>add</action> Will add a new row/record if one does not already exist. If one does exist, an error will be thrown.
    modify

    All tags downloaded can be used to add or modify data.

    Example:<action>modify</action>

    Modify will create a new part or row record if it's not found. If the part/row is found, modify will update the data.
    delete

    All tags downloaded can be used to delete data.

    Example:<action>delete</action>

    Will delete all the column data from a row/record.
    delete_all

    All tags downloaded can be used to delete all of the data.

    Example:<action>delete all</action>

    Will delete all row and column data.
    delete_by_criteria Only available for mass delete of parts based on custom field value.  
    update

    All tags downloaded can be used to update data.

    Example:<action>update</action>

    Update will fail if the row/record is not found. If the record is found, it will modify all the column data in the row.
  3. If you are uploading File Manager files, create an \IMG directory and place all .PDF, .PNG, or .JPG files in that directory.

      Image files must have .PDF, .PNG, or .JPG extensions. Most file types are supported for non-image file attachments.
  4. Zip each individual CSV file and ZIP the \IMG directory.

    ZIP file names can only contain the characters (0-9, a-z, A-Z or _).


ClosedCreating XML Bulk Upload Files

  1. Using a downloaded file as a template, create an XML file containing the records you want to change.

    We recommend that you create a different file for each data type.

      All values, including leading and trailing spaces, are trimmed by the system.

      The default value for all fields with a specified value is an empty string.
  2. In the Action column, for each row of data, enter an Action type from the table below.

    Upload Category Category Name Add Update* Modify Delete Delete All
    Account/User Associations bm_crm_user_assoc X X X X X
    Accounts bm_crm X X X X X
    Commerce Processes bm_cm_version X X X    
    Commerce Transaction bm_cm_bs_data   X      
    Configurable Attributes bm_config_att X X X X  
    Configuration Rules bm_config_rules X X X X  
    Data Tables bm_script_data X X X X X
    File Manager bm_shared_file X X X X X
    Models model X X X X X
    Part Extended Description bm_part_extended X X X    
    Parts bm_part X X X X X
    Parts Associations bm_parts_associations X     X X
    Price Book Associations price_book_variable_name X X X X  
    Product Lines product_line X X X X X
    Serial Number Associations bm_part_in_sn X     X X
    Serial Numbers bm_serial_number X X X X X
    User Accounts bm_user X X X    
    Company bm_company X X X    
    Text Administration bm_itn_override X X X    
    Library Functions LIBRARY X X X    
    Product Definition PRODUCTFAMILY X X X    
    Printer Friendly Document DOCENGINE X X X    

    Catalog Data

    CATALOG X X X    
    Configuration Data CONFIGURATION X X X    

    *An Update action pertains to records that already exist.

  3. Confirm that the XML file has the required tags.

    XML Required Tags change every release.

    The easiest way to obtain a current XML template is to download an item of the same category. For more information on downloading data, see Bulk Downloads.

    Tag Tag Function
    <record_list> Starts and ends the file.
    <each_record> Defines the boundaries of a single record (for example, user record, part record, and so on.)
    <category> Determines the type of data included in the record (for example, bm_customer, model, and so on.)
    <action> Informs the system what to do with the record: either add, update, modify, delete, or delete all.
  4. If you are uploading File Manager files, create an \IMG directory and place all the files in that directory.

      Image files must have .PDF, .PNG, or .JPG extensions. Most file types are supported for non-image file attachments.
  5. ZIP each individual XML file and ZIP the \IMG directory.

    ZIP file names can only contain the characters (0-9, a-z, A-Z or _).

ClosedXML File Examples

ClosedRequired Structure for Single XML Record

ClosedRequired XML for a Line Action

ClosedRequired XML for Quote Action


Uploading Files

ClosedUploading Data Files Using the User Interface

  1. ClosedNavigate to the Upload Files List page.
    1. Click Admin to go to the Admin Home Page.
    2. Click - Upload under Bulk Data Services in the Utilities section.

      The Upload Files List page appears.

  1. Click Browse next to the Add File field.

    The File Upload dialog box appears.

  2. Select the ZIP file you wish to upload and click Open.

    You return to the Upload Files List page, with the selected file listed next to the Add File field.

    XML files must be uploaded using FTP. For more information, see the section Uploading Data Using FTP.

  3. If you want to schedule the upload, enter the date and time in mm/dd/yyyy hh:mm format in the Schedule field.
  4. Click Add.

    The file name appears in the bottom half of the page.

    Upload and Delete buttons appear on the bottom right-hand corner of the page.

  5. Do either of the following:
    • To start upload process immediately: In the Select column, click the checkbox corresponding to the filename, and click Upload.
    • To schedule upload at a future time: In the Select column, click the checkbox corresponding to the filename, enter an upload time in the Schedule field, and then click Upload.

    The Upload Status page appears.

  6. Monitor the uploaded task on the Upload Status page; the status is listed in the Status column.

    The task displays Pending until the upload begins. Finished uploads display either Completed or Failed.

    After the bulk upload process finishes, a bulk upload error log is generated containing a summary of the upload process, including a list of error messages. View the error log by clicking log in the Log column for the file record you are interested in.

    The SuperUser receives an email notice when an upload process completes. The automatically generated email contains a summary of the upload, including number of records successfully uploaded and the number of failures.

    Detailed errors only appear in the bulk upload error log.

ClosedUploading Data Using FTP

Use FTP (File Transfer Protocol) to upload files without using the CPQ Cloud User Interface. FTP uploads are available for both .CSV and XML files. For more information on FTP, see FTP Automation.

  1. Create an XML file called upload_list.xml.

It must be this exact file name, or your upload may produce unexpected results.

  1. In the XML file, list all the .ZIP files that will be uploaded, in the order that you want them to be uploaded.
  2. The upload_list.xml file should fill in the <file> nodes with the name of the .ZIP files to be uploaded.

In this example, the only .ZIP file being uploaded is CustomerAddress.ZIP.

Sample Format:

<?xml version="1.0" encoding="UTF-8"?>

<files>

<file>

<name>CustomerAddress.zip</name>

<type>DATA_TABLES_CSV</type>

</file>

<file>

<name>CustomerAddress_Deploy.zip</name>

<type>REGULAR</type>

</file>

</files>

  1. Organize any referenced attachment files into a folder called IMG, and put that folder into the same directory as the XML file.
  2. Zip the attachments into a single .ZIP file.

The file name can only contain the characters (0-9, a-z, A-Z or _).

The file must not contain any path names.

  1. If you have not done so already, open a ticket for Ops to create an FTP profile for you and an automated folder (one that pulls files regularly, on a timer) on the FTP server.

Inform Ops of your application URL and company/client name.

  1. Copy the ZIP files to the FTP location supplied by Ops.
  1. Copy the upload_list.xml file last.

Once this file has been copied, it will execute and upload the ZIP files in the order they are listed.

    If you need to upload the files at a specific time, use Middleware or something similar to schedule the push of the files to the FTP location.
  1. Monitor the uploaded task on the Upload Status page; the status is listed in the Status column.

The task displays Pending until the upload begins. Finished uploads display either Completed or Failed.


ClosedBulk Uploading of CSV Data Tables

CPQ Cloud requires some way to determine which bulk upload .ZIP files should go to. This is determined by a file called upload_list.xml.

You can only bulk upload CSV data tables via FTP.

Create a file called upload_list.xml.

Bulk Uploading of Data Tables is more convenient that using the Import/Export functionality because, with Import/Export through the UI, you can only do one table at a time. With Bulk Uploads, you can include multiple zipped Data Table CSV files within upload_list.xml.

This is typical for weekly updates to Data Table information or daily updates to Data Tables. This allows you to easily bulk update Data Tables.

For the DATA_TABLES_CSV type, the ZIP file name should be the CPQ Cloud folder name that the Data Table is located in. The name of the CSV file within the ZIP file is the actual table name.


ClosedViewing the Upload Status

You can view the status of all upload tasks scheduled during the previous week.

Successful uploads with Completed in the Status column are removed from the Upload Status page every 7 days. Upload tasks with any status other than Completed are not removed from the Upload Status page.

  1. Navigate to the Upload Status page.
    1. Click Admin to go to the Admin Home Page.
    2. Click (Status), next to - Upload, under Bulk Data Services in the Utilities section.

      The Upload Status page appears.

Using this page, you can:

  • Click Refresh to update the data on the page.
  • Sort the details based on Status, Status Message, Scheduled Date/Time, and Log.

Click the appropriate heading to sort the details in the order you want to view them in.

  • View error logs for a file by clicking log in the Log column for that row.

This displays a new window with all the log entries for the errors and warnings resulting from the upload process selected.

  • To stop a currently running upload, check the check box for the upload filename you want to stop, and click Stop Running Task. Uploads with a status of Running can be stopped.
  • Remove listings for old uploads by selecting the files to be deleted and clicking Delete. Uploads with a status of Pending or Completed can be deleted from the Upload Status page.

To prevent an upload from happening in the future, check the check box for the upload filename you want to cancel, and click Delete.

    If an upload fails, retry the process. If you cannot get a upload to succeed, contact an implementation engineer.

    If you want to wait for an upload process to complete, click Refresh to refresh the content of the Upload Status page.
    Data already uploaded into the system is not saved when an upload process is stopped. It will not be added to the database.

    To add the data, re-run the upload process.

EXAMPLES

ClosedDeleting Parts by Criteria

You can delete multiple parts by XML upload based on Parts Field Values. Use the delete_by_criteria action in the upload XML to specify the criteria for which the parts must be deleted.

The criteria is defined by specifying the part field, the value of the field, and the comparator with which the comparison should be made. When the criteria is satisfied, the part is deleted.

ClosedParts Fields Available for Use

ClosedAllowed Comparators

ClosedSample XML: delete_by_criteria Action

ClosedSample XML: Like Comparator


ClosedDeleting Parts with Multiple Criteria

Multiple criteria can be declared within each_record tag. The criteria will have an implicit AND between them.

To achieve a criterion's OR, use multiple each_record tags and specify the OR criteria in different each_record tags.

Example: You want to delete all parts where _part_custom_field2 = "Domestic" and lead_time > 5. Upload an XML file where the delete_by_criteria action allows the user to specify the two criteria for which the parts must be deleted.

ClosedSample XML: Deleting Parts by Multiple Criteria


TROUBLESHOOTING

ClosedTroubleshooting

After the bulk upload process finishes, a bulk upload error log is generated containing a summary of the upload process, including a list of error messages. Your SuperUser also receives email notice when an upload process occurs. The automatically generated email contains a summary of the upload, including number of records successfully uploaded and number of failures.

Detailed errors only appear in the bulk upload error log.


ClosedReading Error Logs

The Upload error log reports errors that occur during the bulk upload process. The upload error log header contains the number of items uploaded successfully, the number of items that experienced some type of failure, and the time it took for the upload process to complete. Specific errors appear below the log header.

Error messages are severe in that they prevent specific records from being uploaded. When errors occur, the log file reports the failed record's data and a reason for the error.

Warning messages display below the list of error messages. The difference between a warning message and an error message is its severity. Warnings call out minor errors that do not prevent an item from being uploaded.

Example: If a part references an image file that is not included in the ZIP file, the part will be uploaded, but the image will not be associated with it.

Similar to error messages, warning messages contain information to help you pinpoint the problem, such as the line number of the record containing the warning, the cause of the warning, and a suggested solution.

ClosedHow do I correct a problem reported in an upload error log?

You can correct error log warnings and errors in several ways:

The method you choose to correct error log problems should take into consideration the number of problems that need to be fixed. If many problems need to be fixed, then manually entering the information could be a time consuming process. In such cases, it is often quicker to fix the errors directly in the XML files and then re-upload everything.

ClosedWhy are so many errors logged?

The error log is there to help in debugging problems. Every time an "illegal" input is applied to a field, the error is caught by a lower level of the CPQ Cloud software and propagated up to the surface before it is displayed to the user.

Example: if a user were to enter the string "Two" where a number was expected, an error would be caught on the server side and an error was would be passed back along with the stack trace, the nested calls to lines of code within the server-side of CPQ Cloud.

This is all very complex and meant to be kept transparent. The end result of this is an error message like "Encountered " ")" ") "" at line 1, column 81. Was expecting one of:

<NUMBER> ...

<STRING> ..."

to be displayed towards the top of the window. This is mirrored, along with the stack trace, in the error log.

If a user were to use trial and error, and enter "Two", then "two", then "TWO", then "2", 3 errors with stack traces would be produced, adding up to possibly 500 lines in the error log.


NOTES

Each field must contain the data type the system expects.

For example, you can upload new pricing information for parts and new pricing rules for models at the same time. Or you can add new user accounts and CRM customer records in the same XML file.

Each individual category will be defined within each record in the XML, meaning:

<each_record>

<category>bm_cm_bs_data/category>

<action>update</action>

 

... Other XML ...

 

</each_record>

<each_record>

<category>bm_part</category>

<action>update</action>

 

... Other XML ...

 

</each_record>

<each_record>

<category>bm_crm</category>

<action>modify</action>

 

... Other XML ...

 

</each_record>

A data upload is much simpler if each data type's changes are in a file specific to that data type. We recommend using different files for each data type.

The more records included in an upload file, the longer it takes for the process to complete. However, individual records are visible on the system as soon as they are uploaded.

To use the delete action for a File Manager Bulk Upload process, you must have either the file name or the folder name. A Delete action does not work if you only use tags as a description.

RELATED TOPICS

Related Topics Link IconSee Also