Application Development, Product to Market
  • Home
  • Notes
  • Projects
  • Resources
    • Discovery Log
    • Books I Read
    • Blogs I Visit
    • Tools I Use
  • Home
  • Notes
  • Projects
  • Resources
    • Discovery Log
    • Books I Read
    • Blogs I Visit
    • Tools I Use

Copy Marklogic Data between Marklogic Servers - XQsync

10/2/2013

0 Comments

 
Picture
For Marklogic version below 6.0 on different platforms, XQSync is a tool that can copy documents and their metadata between databases,  package documents and their metadata as zip archives, or write them directly to a filesystem. It deals with an entire database, a collection, a directory, or the results of evaluating an XQuery expression.  If you're using Marklogic version 6.0 and above, the way to do this is to use a new Marklogic supported tool named mlcp.

Install XQsync

XQsync is a Java program. First, download dependency JARs:
  1. Download xqsync.jar.
  2. Download the latest MarkXCC.jar (marklogic-xcc-6.0.4.jar) here.
  3. Download XStream (xstream-1.4.5.jar) here.
  4. Download the latest XPP3 (xpp3-1.1.3_8.jar) here.
  5. Move/Put the above xqysync.jar, marklogic-xcc-x.x.x.jar, xstream-x.x.x.jar and xpp3-1.1.3_8.jar to under the same <xqy-installation> directory.
Then, test it out,
cd <xqy-installation>
java -cp xqsync.jar:marklogic-xcc-6.0.4.jar:xstream-1.4.5.jar:xpp3-1.1.3_8.jar \
 -Dfile.encoding=UTF-8 \
 com.marklogic.ps.xqsync.XQSync
You should get the following error about the source Marklogic is not specified:
Exception in thread "main" java.io.IOException: missing required property: INPUT_CONNECTION_STRING
You should be good to proceed.

Export Marklogic Data

To export data from a source Marklogic, the target Marklogic need need an XDBC server to allow XML Content base Connector (XCC) applications (e.g. XQsync) to communicate with MarkLogic Server. Assuming you the source Marklogic already has one setup, the following command will export data to a .zip file:
java -cp xqsync.jar:marklogic-xcc-6.0.4.jar:xstream-1.4.5.jar:xpp3-1.1.3_8.jar \
 -Xmx100m \                           // 1
 -DINPUT_CONNECTION_STRING=xcc://<user>:<password>@<server>:<port>/<database> \
 -DINPUT_DIRECTORY_URI=/content/cq \             // 2
 -DOUTPUT_PACKAGE=/tmp/cq-content.zip \          // 3
 -Dfile.encoding=UTF-8 \
 com.marklogic.ps.xqsync.XQSync
[1] In case heap memory size is not enough.
[2] Limit export to a subset (directory) of the whole database.
[3] Output file name. The actual output file will be /tmp/cq-content-000.zip

Import Marklogic Data

To import data into a target Marklogic, the target Marklogic need need an XDBC server to allow XML Content base Connector (XCC) applications (e.g. XQsync) to communicate with MarkLogic Server. To set one up, use the MarkLogic Server admin interface, which runs on port 8001. 
http://<server>:8001
Click on Configure > App Servers > Create XDBC, and fill in the following values:

  • server name: 9000-Docs                  // 1
  • root: /tmp                              // 2
  • port: 9000
  • modules: (filesystem)
  • database: Documents                      
[1] XDBC server display name. 
[2] Module directory root.

Once the XDBC server is ready, the following command imported the exported file into the target Marklogic's 'Documents' database:
java -cp xqsync.jar:marklogic-xcc-6.0.4.jar:xstream-1.4.5.jar:xpp3-1.1.3_8.jar \
 -Xmx100m \
 -DOUTPUT_CONNECTION_STRING=xcc://admin:admin@localhost:9000/Documents \
 -DINPUT_PACKAGE=/tmp/cq-content-000.zip \
 -Dfile.encoding=UTF-8 \
 com.marklogic.ps.xqsync.XQSync
For debugging purposes, you may tail the Marklogic log to see what's going on as it goes. On my Mac,
cd ~/Library/Application Support/MarkLogic/Data/Logs
tail -f *
URI privileges are stored in the security database, as are exec privileges, roles, and users. None of these are copied by xqsync. So if the imported documents has roles (e.g. ml-read) not yet exist in the target Marklogic, the log may spit out errors:
==> ErrorLog.txt <==
2013-10-02 17:23:39.434 Notice: 9000-Docs: SEC-ROLEDNE: Role does not exist: sec:role-name = ml-read
It would be possible to use xqsync to export your Security database to fix the above missing user, but importing to Security database would be somewhat tricky. I'd suggest you to first manually create the required missing roles (via Configure > Security > Roles > Create ...) in the target Marklogic and re-import again. On success, the output of the import will look like:
>>> ...
>>> setting properties for /content/cq/public/employment/losing-a-job/who-can-get-unemployment-benefits.xml
>>> setting properties for /content/cq/courts/598.xml
2013-10-02 17:51:58.904 INFO: exiting after 1077/1077, 78 events/s, 254 kB/s
2013-10-02 17:51:58.905 INFO: closing factory
2013-10-02 17:51:58.905 INFO: completed 1077 in 13798 ms (78 docs/s)

References

  • XQsync
  • XQSync: a Wheelbarrow for Content
0 Comments



Leave a Reply.

    Categories

    All
    Algorithm
    Concurrency
    CQ
    Data Structure
    Design Pattern
    Developer Tool
    Dynamic Programming
    Entrepreneur
    Functional Programming
    IDE
    Java
    JMX
    Marketing
    Marklogic
    Memory
    OSGI
    Performance
    Product
    Product Management
    Security
    Services
    Sling
    Social Media Programming
    Software Development

    Feed Widget

    Archives

    May 2020
    March 2020
    April 2018
    March 2018
    February 2018
    December 2017
    March 2017
    November 2016
    June 2016
    May 2016
    April 2016
    October 2015
    September 2015
    August 2015
    September 2014
    July 2014
    June 2014
    May 2014
    March 2014
    January 2014
    December 2013
    November 2013
    October 2013
    September 2013
    August 2013
    July 2013
    June 2013

    RSS Feed

in loving memory of my mother  and my 4th aunt
Photos used under Creative Commons from net_efekt, schani, visnup, Dan Zen, gadl, bobbigmac, Susana López-Urrutia, jwalsh, Philippe Put, michael pollak, oskay, Creative Tools, Violentz, Kyknoord, mobilyazilar