Friday, January 27, 2012

How to fix Out Of sync cluster issue CQ / CRX / WEM / ADEP

Use case : Your cluster is out of sync or you get following error in log

27.01.2012 14:09:12 *WARN * ClusterTarSet: Could not open (, line 710) This cluster node and the master are out of sync. Operation stopped.
Please ensure the repository is configured correctly.
To continue anyway, please delete the index and data tar files on this cluster node and restart.
Please note the Lucene index may still be out of sync unless it is also deleted.
Repository home: /crx-quickstart/crx.0000, workspace: crx.default,
file name: data_00037.tar, position:13252096, expected: 109, got: 116,
last data file in cluster node: 37


*INFO * ClusterTarSet: close /local/dss/apps/crx-quickstart/crx.0000 crx.default (, line 408)
27.01.2012 14:18:11 *ERROR* RepositoryImpl: Failed to initialize workspace 'crx.default' (, line 540)
javax.jcr.RepositoryException: Cannot instantiate persistence manager
at org.apache.jackrabbit.core.RepositoryImpl$WorkspaceInfo.initialize(
at org.apache.jackrabbit.core.RepositoryImpl.initStartupWorkspaces(
at org.apache.jackrabbit.core.RepositoryImpl.(
Caused by: This cluster node and the master are out of sync. Operation stopped.
Please ensure the repository is configured correctly.
To continue anyway, please delete the index and data tar files on this cluster node and restart.
Please note the Lucene index may still be out of sync unless it is also deleted.

1) Stop your slave (Your slave will not start anyway)
2) Make sure that You have latest CRX Hot fix or Hotfix greater than installed on your system (Master). If not please contact Adobe to get latest HF. (In order to check for Hot Fix, Go to host:port/crx/index.jsp and you should see crx hotfix version name (For example on top)
3) Once you have latest HF. Install HF in your master and restart master. If you have more than 2 Node cluster then install HF in all other nodes.
4) Once master is up take online backup of master instance (
4.1) if You already have master with latest HF, You can use any valid backup of master (May be backup from last night).
4.2) If you can afford downtime, then take offline backup of crx-quickstart folder from master.
5) Copy backup over to slave instance
6) Go to crx-quickstart/repository/ file and add IP address of master and slave. You can use command like
echo "addresses=MASTER_IP_ADDRESS,SLAVE_1_IP_ADDRESS" >> crx- quickstart/repository/
7) remove crx-quickstart/repository/ file
8) If your master had preferredMaster setting in repository.xml, Remove that setting as well from crx-quickstart/repository/repository.xml file.
9) Delete logs folder crx-quickstart/logs
10) Start slave
11) Based on how much changes has been made, After backup is taken, Your slave take some time to start.

It is important that you exercise following order while stopping and starting,
1) While stopping
-- Stop Slave first and then master
2) While Starting
-- Start last master first and then slave
To identify which one was last master please check

If any one of the node in cluster is running and you have latest Hot Fix, You can stop other nodes in any order.


Tuesday, January 24, 2012

How to make http and https version of site in CQ / WEM

Use case : You want to access your site using both http and https protocol and at same time want all mapping in /etc/map to respect protocol as well. For example if requested page has protocol http then /etc/map/http should get preference and for https /etc/map/https should get preference.

Why : External user can only access https but internal can access http as well or some crazy reason :D

Pre requisite : CQ5.5 or 5.4 with some HF.

Assumption :
Publish http running on 4503
publish https running on 8443
web server http running on 80
web server https running on 443
Goal here is stabilise communication between webserver http to publish http and webserver https to publish https

Refer for how to set up all these.

Approach : https communication between dispatcher and publish is not possible (I mean https port of dispatcher communicating to https port of publish). So you have to create a SSL tunnel (stunnel) and then communicate through that. At a same time you need two cache to cache https and http site separately (Why ?? because links in https sites are rewritten to https and to http site in http, And you want to cache them accordingly, If you don't care about this then there is no need for this step).

Solution :
Step 1: Creating two separate cache for http and https site.
in httpd.conf (Or your embedded conf file) have two separate vhost entry for both http and https and use different document roots for them.
So your httpd.conf will look like this,

<VirtualHost *:80>
ServerName localhost
DocumentRoot <Path for http cache>
... All of your configuration
<Directory "<path for http cache>">
... All configuration

Listen 443
<VirtualHost *:443>
ServerName localhost
DocumentRoot <Path for https cache>
... All of your configuration
#this is responsible for forwarding SSL request to another farm
<Location />
RequestHeader set Host ""

<Directory "<path for https cache>">
... All configuration

Then in your dispatcher.any file create two separate farm as,

... All your configuration
# the load will be balanced among these render instances
/hostname "localhost"
# port of the render (This is stunnel port which we set on step 2)
/port "8081"
... configurations
/docroot "<Https cache docroot>"
.. All your farm 1 configuration

#----------- Farm 2

# first farm entry (label is not important, just for your convenience)

... All your configuration
# the load will be balanced among these render instances
/hostname "localhost"
# port of the render (This is stunnel port which we set on step 2)
/port "4503"
.. All your farm 2 configuration
/docroot "<Http cache docroot>"
.. All your farm 2 configuration

Step 2: Enable stunnel on dispatcher server (Assuming dispatcher server is sun solaris, for different server approach would be different)

(On the dispatcher server) Run the following commands to install stunnel
1) sudo yum install stunnel
2) sudo /sbin/chkconfig --add stunnel
3) Open /etc/stunnel/stunnel.conf for editing using this command sudo vi /etc/stunnel/stunnel.conf
4) In stunnel.conf set client = yes
5) Add this to stunnel.conf (replace <cq5-instance-hostname> with the hostname of the cq5 instance)
accept = 8081
connect = :8443
6) start stunnel
7) sudo service stunnel start
8) Configure CQ dispatcher to point to stunnel port instead of the publish instance (Which is already done, See above)
9) restart apache and verify that the dispatcher still works when going through stunnel

(If you are using mac you can do following to install and start)
1) download build from
2) untar
3) Using command line do following { Make sure that X-code is installed for this to work }
sudo make install
stunnel will get install under /usr/local/etc/stunnel/
4) go to /usr/local/etc/stunnel
5) sudo cp stunnel.conf-sample stunnel.conf
6) sudo vi stunnel.conf
7) Add following property (Or change them)
setuid = root
setgid = wheel
debug = 7
output =
debug = local6.err
;this is require to check if there is any error. Please comment this to run this process in foreground
foreground = yes
cert = <path for valid cert> {Give valid cert and key, Make sure that permission is valid}
key = <path for valid key>
;Comment everything and add following entry
; Example CQ instalce connection
client = yes
;Accept on this port {This is what you configure on dispatcher.any}
accept = 8081
;Redirect to this port {This is CQ port running on https}
connect = 8443
8) Start stunnel using command
sudo stunnel stunnel.conf
9) Configure CQ dispatcher to point to stunnel port instead of the publish instance (Which is already done, See above)
10) restart apache and verify that the dispatcher still works when going through stunnel

Now you should be all set to communicate from https port of dispatcher to https port of publish.

Special thanks to Adobe team member for guidance.

Important note: I have not tested this completely. Please test it before using it in production. And as always please read disclaimer :)

Tuesday, January 17, 2012

How to delete packages created before certain date from package manager using curl in CQ / WEM

Use case: You upload CQ packages a lot and end up having many packages in package manager. This cause package manager to load slow, also cause un necessary space consumption.

Solution: You can use following unix script to delete packages created before (Or after) certain date.

Please change variable based on your need.

I have commented curl command for now, First see if right package is getting deleted then uncomment curl command to delete.


# Author:

# The host and port of the source server


# The user credentials on the source server (username:password)


#This is your package location in CRX. Change this package location based on your need


# This is date after which you want to delete all package <YYYY>-<MM>-<DD>



#Query to get all files uploaded after certain date

ALL_PATHS=`curl -s -u $SOURCE_CRED "$SOURCE/bin/querybuilder.json?path=$PACKAGE_LOCATION&type=nt:file&p.limit=-1&

echo "$ALL_PATHS"



# I have commented this line. Please see what is getting deleted before deleting it

# curl -u $SOURCE_CRED -X POST http://$SOURCE/crx/packmgr/service/.json$SINGLE_PATH?cmd=delete


You can also do if else within script to compare package name

if [[ $string =~ .*My.* ]]




Please check Packages on the Command Line for different curl options

IMPORTANT NOTE: Please test this script before use. I have not tested this script yet :)

How to use and debug CQ Client Library CQ5.4 / WEM

Use case:
1) You want to create your own client library (Only basic).
2) You want to debug performance of javascript
3) You want to list all client library
4) You want to have all your js and css in same location {No need to have it under /etc/design}


To create your own client library

You can use structure like,

+ /apps/yourapp/ui [cq:Folder]
+ clientlib [cq:ClientLibrary]
- categories ["cq.widgets", "cq.wcm.edit", "cq.admin",<Or any category>]
- dependencies []
- type "js"
- js.txt [nt:file]
- channels []
- embed []
+ include.list [nt:file]
+ exclude.list [nt:file]
+ files [cq:Folder]
+ MyCustomeJs1.js [nt:file]
+ MyCustomeJs2.js [nt:file]
+ themes [cq:Folder]
+ default [cq:ClientLibrary]
- categories ["cq.widgets", "cq.wcm.edit", "cq.admin",<Or any category>]
- dependencies []
- type "css"
- css.txt
- channels []
- embed []
+ include.list [nt:file]
+ exclude.list [nt:file]
+ files [cq:Folder]
+ MyCustomeCSS1.css [nt:file]
+ MyCustomeCSS2.css [nt:file]

Additional descriptions of the cq:ClientLibrary properties (Information can not be 100% correct):

categories: List of tags or dynamic dependencies. If a category is used in the ClientLibraryManager.include() method, all cq:ClientLibrarys tagged with this category will be included in the html. See example below.
dependencies: A list of either categories or paths to cq:ClientLibrarys that have to be included in html before (and will be included automatically).
type: "css" or "js", hence <link rel='stylesheet'> or <script> in the html
includes.txt list file defines the concatenation order
exclude.txt That should not be included for concatenation
- if no channels are specified, it's only assigned to all channels
- if multiple channels are specified, it's assigned to all those channels
- special keyword default can be used to specify the default channel
- if the channel name is prefixed with !, it is not used on that channel.
// all channels
channels: []
// only default channel
channels: ["default"]
// only on touch devices (/libs/cq/touch/redirect/notimplemented, /etc/designs/geometrixx/clientlibs/ie6/themes/default)
channels: ["touch"]
// not on touch devices (/etc/clientlibs/foundation/swfobject)
channels: ["!touch"]

Use Case:
touch devices
create a channel specific lib: cq.myWidget, channels=["touch"]
exclude that channel in the normal lib: cq.myWidget, channels=["!touch"]
create a channel detection script based on user agent
use it: <cq:inclideClientLibrary categories="cq.myWidget"/>

See one more example

/etc/designs/geometrixx/clientlibs/ie6/themes/default which is included /apps/geometrixx/components/asseteditor/headlibs.jsp

A library can embed other libraries. This is useful to reduce the amount of includes or to bypass some access restrictions of libraries that reside in a read-protected area. For example the clientlibs of the foundation components are located in /libs/foundation/* but are needed on publish (which has /libs read protected for anonymous).
When a library defines categories to be embedded, if only embeds the direct dependencies that match the same type (obviously) and theme.
Example, /etc/designs/geometrixx/clientlibs has,

allowProxy: (CQ5.6 Onward) true or false
The purpose is to serve client libs hosted under /libs and /apps
  • add the property: allowProxy = 'true' to a clientlib
  • the clientlib will then be proxied via /etc.clientlibs/..., bypassing access control set on the clientlib.
Assume you have clientlib in /apps/myproject/clientlibs/foo. then you set the property /apps/myprojects/clientlibs/foo/allowProxy to true and then you can request: /etc.clientlibs/myprojects/clientlibs/foo.js.

How to use custom client library

the libraries can either be included directly in the html,
<link rel="stylesheet" href="/libs/cq/ui/themes/default.css" type="text/css">
<script type="text/javascript" src="/libs/cq/ui/widgets.js"></script>

or via manager. eg:
ClientLibraryManager.include(out, "/libs/cq/ui/widgets");

ClientLibraryManager.include(out, "wcm.edit");

or using the tag library:
<cq:includeClientLib categories="cq.packaging"/>

using the manager or tag is of course favorable since if the ClientLibraryManager runs in debug mode, the concatenation is disabled.
To avoid duplicated inclusions when multiple libraries with dependencies are requested, the include method supports multiple paths or categories. Multiple requested libraries (through the API or through dependencies) will be only included once:

ClientLibraryManager.include(out, "wcm.edit", "wcm.admin", "wcm.edit");

How to Debug custom client library

By Using ?debugClientLibs=true (on a html page). This will load individual javascript and then you can use request analyzer to analyze each js.
By using ?debugConsole=true (on a html page) if you are using firebug

Since 5.4, there is a overview page available that lists all libraries and allows to test resolution.

See a good example here

One more very good blog Post

If you want to further minify js/css you can use "Day CQ HTML Library Manager" from felix configuration

Note This information is not a official documented information. Please test it before use.

Special thanks to Adobe and Alex from Adobe for providing this information.

Thursday, January 12, 2012

How to reduce Lucene Index size in CQ / WEM

Use Case:
1) Index size is huge and it is taking a lot of time for CQ to start.
2) Index size is huge and it is taking a lot of time for Index to rebuild (Also check for this).
3) you use different search service like solr or FAST for indexing
4) you don't use full text searching of documents in site


Step 1: Find tika_config file

For version CQ5.4 and less

If you are really not concern about full text indexing of these documents, You could disable indexing of these document in tika-config (crx-quickstart/server/runtime/0/_crx/WEB-INF/classes/org/apache/jackrabbit/core/query/lucene/tika-config.xml). If this folder structure is not present then you have to create one. Original tika_config.xml can be found by unzipping crx-quickstart/server/runtime/0/_crx/WEB-INF/libs/jackrabbit-core-*.jar (Copy it to some other location, rename it to .zip and then unzip) and then going to org/apache/jackrabbit/core/query/lucene).

For version CQ5.5

1) Find the jackrabbit-core jar file and extract the tika config: find ./crx-quickstart/launchpad/felix -name "jackrabbit-core*.jar" | xargs -I {} jar -xvf {} org/apache/jackrabbit/core/query/lucene/tika-config.xml

2) Update org/apache/jackrabbit/core/query/lucene/tika-config.xml file with updated tika file (See step 2)

3) Update the jackrabbit-core jar file with the updated tika-config.xml file: find ./crx-quickstart/launchpad/felix -name "jackrabbit-core*.jar" | xargs -I {} jar -uvf {} org/apache/jackrabbit/core/query/lucene/tika-config.xml

Step 2: Modify file

You could add org.apache.tika.parser.EmptyParser as class for not to parse document type.

For example (To not index excel sheet)

<parser class="org.apache.tika.parser.EmptyParser">

To remove PDF parsing you can remove entry
<parser name="parse-pdf" class="org.apache.tika.parser.pdf.PDFParser">

Above method will also help you to reduce Index size (Lucene) in CQ.

Note: To reduce Lucene Index size you can also add following in workspace.xml

<SearchIndex class="">
<param name="path" value="${wsp.home}/index"/>
<!-- add below param -->
<param name="supportHighlighting" value="false"/>

As an example please refere this tika-config.xml file.

Some use ful link for tuning your search index in case you can do above,
try to tune "resultFetchSize" and other parameters

Step 3: Disable Indexing using indexing_config.xml file 

Please check instruction below of how to do that. You can add your own node type to reduce index size further. You can use attached indexing_config file

Other Useful Links to reduce lucene Index:

There is an also ongoing issue to expedite start up time if index size is large

Important:You have to rebuild index after above changes.

Special thanks to Andrew Khoury and other member from Adobe for sharing information.

Tuesday, January 10, 2012

How to change different log locations in CQ {Till CQ5.4}

Use Case: You want to consolidate all logs to one location

Solution: You can change log locations of various logs in CQ by changing following files

Note that Whole CQ (Till CQ5.4) is composed of three different application (Which you can see under <host>:<port>/admin,
And each of them has different logging in corresponding logs folder.

CRX/CQ -- Under /crx-quickstart/logs
CQSE -- under /crx-quickstart/server/logs -- You change this location through server.xml (see for log-file-name value)
launchpad -- under /crx-quickstart/launchpad/logs -- You change this location under /crx-quickstart/launchpad/ (see for value)
/crx-quickstart/server/logs/startup.log file location is defined under /crx-quickstart/server/serverctl file. Look for CQ_LOG="${CQ_LOG:-"$CQ_LOGDIR/startup.log"}". Note that you can override this in start script as well.

Please refer to set up custom log location for Sling or CRX

Other logs which should not touched

/crx-quickstart/repository/repository/index/redo.log -- Used for indexing. Never change this location.

/crx-quickstart/repository/shared/journal/journal.log -- For upgraded version, This is used for clustering. Don't change this location as well.

/crx-quickstart/repository/repository/revision.log -- Used for clustering synchronization. Don't change location of this log.

Wednesday, January 4, 2012

How to set up run mode in CQ / WEM

Run modes allow you to tune your CQ instance for a specific purpose; for example author, publish or development. This is done by defining collections of configuration parameters for each run mode. A basic set is applied for all run modes, additional sets are each tuned to the purpose of your specific environment.
All configuration settings are stored in the one repository and activated by setting the Run Mode.
Standard run modes are:

• author
• publish
You can also create environment specific run mode such as,

• author, development
• publish, test
• author, intranet, us
• as required...

There are two mechanisms for setting standard and environment specific run mode for your instance:
To set up standard run mode Use the naming convention:
For example, set a standard run mode by naming the jar file cq-author-4502 or cq-publish-4503

To set up environment specific run mode there are two methods,
1) Through <cq-installation-dir>/crx-quickstart/launchpad/
Add the following properties (following example is for author, test, uk):*/(install|config)?,test,uk
In above case will get picked up (Or whatever with maximum match)

2) Through <cq-installation-dir>/crx-quickstart/launchpad/ and system property (In start script):*/(install|config)? (In sling property, this is because there is bug in CQ that by default this regex is set up for author),prod,us (In start script)
In above case will get picked up (Or whatever with maximum match)

Configuration values for the run modes are saved in the repository. You can store all configurations in one repository as the run mode is indicated by a suffix on the folder name; for example:
• config, applicable for all run modes
•, used in author run mode
• config.publish, used in publish run mode
• config.<standard-run-mode>.<environment-specific-mode>, used in the applicable run mode

Following order of precedence are used to set up run mode (From top to bottom, Top being highest)
1) From jar file
2) From Sling property
3) From System property

So in above case jar file name will have precedence over system property.

Note that from CQ5.5 there is no need to set sling property regexp to set up run mode. Setting run mode through system property will be enough to decide standard run mode (As regexp issue is fixed in CQ5.5). Another workaround you can do for run mode to work OOTB in CQ5.4 without regexp is use run mode as instead of

How to check run mode

How to read config using API
Method 1
@Reference(cardinality = ReferenceCardinality.MANDATORY_UNARY, policy = ReferencePolicy.STATIC)
private ConfigurationAdmin configAdmin;

protected void activate(ComponentContext ctx) {
Configuration conf = configAdmin.getConfiguration("your configuration pid");
String myProp = (String) conf.getProperties().get("any property in your configuration");

Method 2
import org.osgi.framework.BundleActivator;
import org.osgi.framework.BundleContext;
public class Activator implements BundleActivator {
private ConfigurationAdmin confAdmin;
public void start(BundleContext context) throws Exception {
this.confAdmin = (ConfigurationAdmin) context.getService(context.getServiceReference(ConfigurationAdmin.class.getName()));
Configuration conf = this.confAdmin.getConfiguration("your configuration pid");
String property = (String) conf.getProperties().get("property");

Method 3, Using configuration impl
import java.util.Dictionary;
import org.osgi.service.component.ComponentContext;

protected void activate(ComponentContext componentContext) {
Dictionary<?, ?> configuration = componentContext.getProperties();
String value = (String) configuration.get(name);

Needless to say that your bundle is registered as component

Method 4, When I don't have access to bundle or component context

private static final String MY_SERVICE_CLASS_NAME = MyService.class.getName();

private MyService getMyService() {
Bundle bundle = org.osgi.framework.FrameworkUtil.getBundle(this.getClass());
BundleContext bundleContext = bundle.getBundleContext();
final ServiceReference ref = getBundleContext().getServiceReference(MY_SERVICE_CLASS_NAME);
if(ref == null) {
String message = "No OSGI Services registered under '" + MY_SERVICE_CLASS_NAME + "'";
throw new IllegalStateException(message);
return (MyService)bundleContext.getService(ref);

Thanks Wei Zhang from Adobe for providing this method.

Reference: Modes

Please note that if no standard mode is specified CQ assumes it is author mode through quickstart.

Special thanks to Carsten Ziegeler from Adobe.