Copy Marklogic Data between Marklogic Servers - XQsync

For Marklogic version below 6.0 on different platforms, XQSync is a tool that can copy documents and their metadata between databases,  package documents and their metadata as zip archives, or write them directly to a filesystem. It deals with an entire database, a collection, a directory, or the results of evaluating an XQuery expression.  If you're using Marklogic version 6.0 and above, the way to do this is to use a new Marklogic supported tool named mlcp.

---

Install XQsync

XQsync is a Java program. First, download dependency JARs:

1. Download xqsync.jar.
   
2. Download the latest MarkXCC.jar (marklogic-xcc-6.0.4.jar) here.
   
3. Download XStream (xstream-1.4.5.jar) here.
   
4. Download the latest XPP3 (xpp3-1.1.3_8.jar) here.
5. Move/Put the above xqysync.jar, marklogic-xcc-x.x.x.jar, xstream-x.x.x.jar and xpp3-1.1.3_8.jar to under the same <xqy-installation> directory.

Then, test it out,

cd <xqy-installation>
java -cp xqsync.jar:marklogic-xcc-6.0.4.jar:xstream-1.4.5.jar:xpp3-1.1.3_8.jar \
 -Dfile.encoding=UTF-8 \
 com.marklogic.ps.xqsync.XQSync

You should get the following error about the source Marklogic is not specified:

Exception in thread "main" java.io.IOException: missing required property: INPUT_CONNECTION_STRING

You should be good to proceed.

Export Marklogic Data

To export data from a source Marklogic, the target Marklogic need need an XDBC server to allow XML Content base Connector (XCC) applications (e.g. XQsync) to communicate with MarkLogic Server. Assuming you the source Marklogic already has one setup, the following command will export data to a .zip file:

java -cp xqsync.jar:marklogic-xcc-6.0.4.jar:xstream-1.4.5.jar:xpp3-1.1.3_8.jar \
 -Xmx100m \                           // 1
 -DINPUT_CONNECTION_STRING=xcc://<user>:<password>@<server>:<port>/<database> \
 -DINPUT_DIRECTORY_URI=/content/cq \             // 2
 -DOUTPUT_PACKAGE=/tmp/cq-content.zip \          // 3
 -Dfile.encoding=UTF-8 \
 com.marklogic.ps.xqsync.XQSync

[1] In case heap memory size is not enough.
[2] Limit export to a subset (directory) of the whole database.
[3] Output file name. The actual output file will be /tmp/cq-content-000.zip

Import Marklogic Data

To import data into a target Marklogic, the target Marklogic need need an XDBC server to allow XML Content base Connector (XCC) applications (e.g. XQsync) to communicate with MarkLogic Server. To set one up, use the MarkLogic Server admin interface, which runs on port 8001. 

http://<server>:8001
Click on Configure > App Servers > Create XDBC, and fill in the following values:

• server name: 9000-Docs                  // 1
• root: /tmp                              // 2
• port: 9000
• modules: (filesystem)
• database: Documents                      

[1] XDBC server display name. 
[2] Module directory root.

Once the XDBC server is ready, the following command imported the exported file into the target Marklogic's 'Documents' database:

java -cp xqsync.jar:marklogic-xcc-6.0.4.jar:xstream-1.4.5.jar:xpp3-1.1.3_8.jar \
 -Xmx100m \
 -DOUTPUT_CONNECTION_STRING=xcc://admin:admin@localhost:9000/Documents \
 -DINPUT_PACKAGE=/tmp/cq-content-000.zip \
 -Dfile.encoding=UTF-8 \
 com.marklogic.ps.xqsync.XQSync

For debugging purposes, you may tail the Marklogic log to see what's going on as it goes. On my Mac,

cd ~/Library/Application Support/MarkLogic/Data/Logs
tail -f *

URI privileges are stored in the security database, as are exec privileges, roles, and users. None of these are copied by xqsync. So if the imported documents has roles (e.g. ml-read) not yet exist in the target Marklogic, the log may spit out errors:

==> ErrorLog.txt <==
2013-10-02 17:23:39.434 Notice: 9000-Docs: SEC-ROLEDNE: Role does not exist: sec:role-name = ml-read

It would be possible to use xqsync to export your Security database to fix the above missing user, but importing to Security database would be somewhat tricky. I'd suggest you to first manually create the required missing roles (via Configure > Security > Roles > Create ...) in the target Marklogic and re-import again. On success, the output of the import will look like:

>>> ...
>>> setting properties for /content/cq/public/employment/losing-a-job/who-can-get-unemployment-benefits.xml
>>> setting properties for /content/cq/courts/598.xml
2013-10-02 17:51:58.904 INFO: exiting after 1077/1077, 78 events/s, 254 kB/s
2013-10-02 17:51:58.905 INFO: closing factory
2013-10-02 17:51:58.905 INFO: completed 1077 in 13798 ms (78 docs/s)

References

• XQsync
• XQSync: a Wheelbarrow for Content