Automate Data Migration Using Salesforce Dataloader CLI

We all have used dataloader to import/export data in/out of Salesforce. There are scenarios where we need to automate it or scenarios where we need to upload different objects in a new Sandbox or Scratch org to make it ready for usage. We can also use SFDX but it can only upload 200 records per command.

Use Case

I had a scenario where after every release we need to refresh a sandbox but also need to make it ready with some data so that testing can be done properly. We can also use Sandbox templates but that, only works with Full Copy or partial copy Sandbox. So the only choice left is Dataloader, but if you have 10+ object and couple of them master-detail and if you have a release every 2-4 weeks and developer sandbox then it becomes very time consuming task every time. So you need to find a way to auto populate all required data in one go. For which we can use Dataloader CLI

Steps to SetUp Dataloader CLI

Assuming you are using windows, either go to data loader installing location (salesforce\Data loder\bin) or add bin folder location to path variable. Now you need to generate an encrypted key for your operation.

.\encrypte.bat -e <text> <keyfile path location>

When data loader is installed a default key is generated in C:\Users\-username-\.dataloader folder with dataloader.key as name. Run the command from Dataloader\bin folder

.\encrypt.bat -e <sometext> C:\Users\--\.dataloader\dataloader.key 

This will generate and encrypted key copy it and save it using notepate with name key.txt in your project folder. I will be using sample object name like Candidate, Position and Job Application in which Position and Job Application are master detail. My project folder looks like this…

First things first, encrypt your Salesfore password including security token with the generated key. Go to Dataloader Bin folder and open command prompt any type .\encrypt.bat -e yoursalesforcepassword C:\—\MyProject\key.txt . If you have proxy then encrypt the proxy password also. Copy and save this passwords because it will be used later

File Explaintations

.sld files – are dataloader mapping files. Which maps the csv column name to salesforce field api name

#Mapping values for Canidate
Name=Name
Date of Birth=DataOfBirth__c
Age=Age__c
Email=Email__c

Key.txt – This file contains the encrypted key generated in first step

config.properties – contains configuration for dataloader. check below example

sfdc.debugMessages=false
process.encryptionKeyFile=C:\\---MyProject\\key.txt
sfdc.endpoint=https\://yourdomain.salesforce.com/
sfdc.username=username
sfdc.password=<Encypted Salesforce Password>
# Proxy info if any
sfdc.proxyUsername=proxy username
sfdc.proxyPassword=<proxy password encrypted>
sfdc.proxyHost= yourproxy.com
sfdc.proxyPort=0000
sfdc.loadBatchSize=200
sfdc.timeoutSecs=600

process-conf.xml – this is the file where you need to specify job for data loader it is a bean file where you have multiple beans to specify each job type

<!DOCTYPE beans PUBLIC "-//SPRING//DTD BEAN//EN" "http://www.springframework.org/dtd/spring-beans.dtd">
<beans>
  <!-- Candidate  -->
  <bean id="candidateInsert" class="com.salesforce.dataloader.process.ProcessRunner" singleton="false">
      <description>candidateInsert inserts caondidate records into Salesforce.</description>
      <property name="name" value="candidateInsert" />
      <property name="configOverrideMap">
          <map>
              <entry key="sfdc.entity" value="Candidate__c"/>
              <entry key="process.operation" value="insert"/>
              <entry key="process.mappingFile" value="C:\-\-MyProject\Candidate\candidate.sdl"/>
              <entry key="dataAccess.name" value="C:\-\-MyProject\Candidate\candidate.csv"/>
              <entry key="dataAccess.type" value="csvRead"/>
              <entry key="process.lastRunOutputDirectory" value="C:\-\-MyProject\Candidate\candidate\output"/>
          </map>
      </property>
  </bean> 
  <!-- Position -->
  <bean id="positionInsert">
      ---
  </bean>
</beans>

process.bat – this is file where we instruct dataloader to execute multiple process configs

echo "Starting"
set projectdir="C:\-\-\MyProject"
cls
cd "C:\Users\-\-\-\-\Data Loader\bin"
CALL process.bat %projectdir% "candidateInsert"
CALL process.bat %projectdir% "positionInsert"
CALL process.bat %projectdir% "jobApplicationInsert"
pause

You can Insert, Update, Upsert and Export data using dataloader cli. For more information check the salesforce dataloader guide or the samples which are available in the dataloader installation folder

Special Case (Master-Detail)

Master detail relations cannot be uploaded directly. For that you need to follow below steps.

  • Create ExternlId field on parent object, map it with old salesforce id or just provide dummy unique values in csv file. Add the mapping in sdl file.
  • In child object sdl mapping file. Taking Position as parent object and JobApplication as child example. In JobApplication sdl file for Position column use Position__r.ExternalId__c=Position__r\:ExternalId__c the column name should also be Position__r.ExternalId__c in csv
  • Make sure Position process is run first in process.bat file

Finally, just double click the process.bat file and wait for the data to be uploaded

One thought on “Automate Data Migration Using Salesforce Dataloader CLI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s