Tuesday, March 20, 2012

How to run online backup / Datastore GC / Tar Optimization using curl in CQ5.5

Use case: Curl command for online backup / Datastore GC / Tar Optimization for CQ5.4 does not work for CQ5.5 any more

Reason: Many CRX related operation is moved to JMX. Now you can use any JMX console or JMX API to invoke these operations.

Solution: You can use following curl commands


For online backup



curl -u <UID>:<PASSWORD> -X POST http://<HOST>:<PORT>/system/console/jmx/com.adobe.granite:type=Repository/op/startBackup/java.lang.String?target=<PATH OF BACKUP>/test.zip


Following options are available for backup

Delay (Default is 10 mili second)

curl -u <UID>:<PASSWORD> -X POST
http://<HOST>:<PORT>/system/console/jmx/com.adobe.granite:type=Repository/a/BackupDelay?value=<TIME-IN-MILISECOND>



Note After running this from front end you will see (null) as response. You have to check CRX log to verify if backup started. In CRX log (/crx-quickstart/logs/error.log) you would see message like this

*INFO* [Backup Worker Thread] com.day.crx.core.backup.Backup Backup started.
*INFO* [Backup Worker Thread] com.day.crx.core.backup.Backup Read size (1.234 gb) in 1.97s


OR Use following CURL for online backup


curl -u admin:admin --data "delay=3&force=false&target=test3.zip" http://HOST:PORT/libs/granite/backup/content/admin/backups/


Where,
delay: Time delay between write block.
force: To override last one
Target: Absolute path of backup. If path is not provided and just file name is given, Then it will use current path of installation.

Once backup is started, You can cancel it using following curl command

curl -u admin:admin http://HOST:PORT/libs/granite/backup/content/admin/backups.cancel.html

You can also track online progress using following URL

http://HOST:PORT/libs/granite/backup/content/admin.html




For Datastore GC


curl -u <UID>:<PASSWORD> -X POST http://<HOST>:<PORT>/system/console/jmx/com.adobe.granite:type=Repository/op/runDataStoreGarbageCollection/java.lang.Boolean


To delete data and delay as 2



curl -u admin:admin -X POST --data "delete=true&delay=2" http://HOST:PORT/system/console/jmx/com.adobe.granite%3Atype%3DRepository/op/runDataStoreGarbageCollection/java.lang.Boolean



CRX Log message to look for

*INFO* [127.0.0.1 [1332343886706] POST /system/console/jmx/com.adobe.granite%3Atype%3DRepository/op/runDataStoreGarbageCollection/java.lang.Boolean HTTP/1.1] com.day.crx.sling.server.impl.jmx.GarbageCollection Scanning /libs/wcm/core/content/siteadmin/actions/create/menu/createPage


Automate DataStore GC (AEM 5.6 onward)

Option 1: Using Configuration

create new configuration name 

<some path for config>/com.day.crx.sling.server.impl.jmx.GarbageCollectionConfig-DatastoreGarbageCollector.xml with following values

<?xml version="1.0" encoding="UTF-8"?>
<jcr:root xmlns:sling="http://sling.apache.org/jcr/sling/1.0" xmlns:cq="http://www.day.com/jcr/cq/1.0" xmlns:jcr="http://www.jcp.org/jcr/1.0"
    jcr:primaryType="sling:OsgiConfig"
    jobName="Datastore Garbage Collector"
    schedule="00 00 23 ? * SUN"
    deleteNodes="{Boolean}true" />

This mean that run Datastore Garbage collection every Sunday at 11 PM.

Option 2: Using CURL and CRON job. Create a Cron job on your machine and run CURL command given above.

 
For Tar optimization

curl -u <UID>:<PASSWORD> -X POST http://<HOST>:<PORT>/system/console/jmx/com.adobe.granite:type=Repository/op/startTarOptimization/


CRX Log Message to look for

com.day.crx.persistence.tar.file.TarFile /crx-quickstart/repository/workspaces/crx.default/data_00000.tar id:0 length:75436032 append:75435008 943288331 optimize: 2420224



Similarly you can use CURL command to different JMX operation listed in

http://<HOST>:<PORT>/system/console/jmx/com.adobe.granite:type=Repository






How to identify master using Curl Command And then run backup only on master

ISMASTER=`curl -s -u admin:admin http://HOST:PORT/system/console/jmx/com.adobe.granite:type=Repository | grep -A 1 crx.cluster.master | sed 's/\<\/*td\>//g' | awk 'NR>1' | awk '{ print $1 }'`
if [ $ISMASTER != 'true' ]; then
echo "Not running backup, that is the slave node";
exit 1;
else
echo "Running online backup this is master Node"
# Your curl command for online backup goes here
fi;

Special Thanks to Andrew Khoury from Adobe for providing this CURL command.


Use firebug to get complete URL and then replace it in above curl command.

9 comments:

  1. Hi Yogesh,

    Sorry for such a stupid question, actually I have not passed the delete parameter in curl command.

    R's
    Rohit Varshney

    ReplyDelete
  2. This comment has been removed by a blog administrator.

    ReplyDelete
  3. Yeah, this is really a good writing skill you have and I appreciate your writing skill. You are share awesome post for people. I think everybody will be happy by read your post.

    ReplyDelete
  4. Absolute path of backup. If path is not provided and just file name is given, Then it will use current path of installation.draas.biz

    ReplyDelete
  5. Hi
    I am trying to create a backup with some time delay. I have set absolute path in the taget as shown in the following command:
    curl -u admin:admin --data "delay=3&force=true&target=/backup/2014" http://localhost:8080/libs/granite/backup/content/admin/backups/ but I don't see any backup file created. Could you please help me with that. I just wanted a backup to a directory and I don't want any zip file.

    ReplyDelete
    Replies
    1. Hello Pavan,

      What response you get when you run this command ? Also does backup works when you do it manually by going to http://localhost:8080/libs/granite/backup/content/admin/backups/ ?

      Yogesh

      Delete
  6. This is first time that I visit here. I found so many exciting matter in this particular blog, One thing I would like to request you that pls keep posting such type of informatics blog.
    werkblad keramiek

    ReplyDelete
  7. Hi,

    I want one instance of 155GB to be backed up.
    What is the best way to take online backup in AEM?
    Which method is faster?

    ReplyDelete
    Replies
    1. I used the following command to take back up.It is taking lot of time.
      "curl -u admin:admin -X POST http://localhost:4502/system/console/jmx/com.adobe.granite:type=Repository/op/startBackup/java.lang.String?target=backup.zip"

      Delete