Wednesday, December 12, 2012

How to remove non referenced node from DAM in CQ 5.5

Use Case: Clean your DAM to remove non referenced node.

Solution: You can use following script to remove all non referenced node from DAM.

# !/bin/bash
# Author: upadhyay.yogesh@gmail.com
# The host and port of the source server
SOURCE="localhost:4502"
# The user credentials on the source server (username:password)
SOURCE_CRED="admin:admin"
#Filter path
ROOT_PATH="/content/dam/<Your PATH>"
ALL_PATHS=`curl -s -u $SOURCE_CRED "$SOURCE/bin/querybuilder.json?path=$ROOT_PATH&type=dam:Asset&p.limit=-1" | tr ",[" "\n" | sed 's/ /%20/g' | grep path | awk -F \" '{print $4 "\n"}'`
echo "$ALL_PATHS"
for SINGLE_PATH in $ALL_PATHS
do
REFERENCE_COUNT=`curl -s -u $SOURCE_CRED "$SOURCE/bin/wcm/references.json?path=$SINGLE_PATH" | tr ",[" "\n" | sed 's/ /%20/g' | grep path | awk -F \" '{print $4 "\n"}'`
if [ "$REFERENCE_COUNT" == "" ] ; then
  echo "Removing $SINGLE_PATH"
  #curl -u $SOURCE_CRED -F:operation=delete $SOURCE$SINGLE_PATH
fi
done

Post Cleaning Step:

1) Run Data store garbage collection see http://www.wemblog.com/2012/03/how-to-run-online-backup-using-curl-in.html or http://helpx.adobe.com/crx/kb/DataStoreGarbageCollection.html for that)
2) Remove these nodes from publish as well if they are not deactivated already (usually deleting activated node should delete it from publish as well, But please check)

Caution: As always please test this. Also this might not cover Image referenced through CSS or your code. This only covers Image referenced through Image or Image related component.

Note: You can also use JSAWK https://github.com/micha/jsawk for better parsing of response.


Monday, November 19, 2012

How to Fix No Such Item repository corruption in CQ5.5

Use Case: You have repository corruption that can not be fixed using consistency check and fix.

Prerequisite:

http://helpx.adobe.com/crx/kb/RepositoryInconsistency.html

http://helpx.adobe.com/cq/kb/cq-5-5--rien-ne-va-plus---i-need-a-text-shell.html

Stack Trace:

05.11.2012 00:30:01.476 *ERROR* [pool-5-thread-2] Unable to perform role import javax.jcr.ItemNotFoundException: failed to build path of 795cff24-7fd7-3f5f-a9b1-4724c3a6ff4d
        at org.apache.jackrabbit.core.HierarchyManagerImpl.getPath(HierarchyManagerImpl.java:400)
        at org.apache.jackrabbit.core.CachingHierarchyManager.getPath(CachingHierarchyManager.java:233)
        at org.apache.jackrabbit.core.ItemImpl.getPrimaryPath(ItemImpl.java:188)
        at org.apache.jackrabbit.core.NodeImpl.getPrimaryPath(NodeImpl.java:2753)
        at org.apache.jackrabbit.core.ItemImpl$2.perform(ItemImpl.java:379)
        at org.apache.jackrabbit.core.ItemImpl$2.perform(ItemImpl.java:376)
        at org.apache.jackrabbit.core.session.SessionState.perform(SessionState.java:216)
        at org.apache.jackrabbit.core.ItemImpl.perform(ItemImpl.java:91)
        at org.apache.jackrabbit.core.ItemImpl.getPath(ItemImpl.java:376)
        at <Some Custom Code>
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.jackrabbit.core.state.NoSuchItemStateException: 34228aff-f648-429a-a097-29de8568fe3e
        at org.apache.jackrabbit.core.state.SharedItemStateManager.getItemState(SharedItemStateManager.java:285)
        at org.apache.jackrabbit.core.state.LocalItemStateManager.getNodeState(LocalItemStateManager.java:110)
        at org.apache.jackrabbit.core.state.LocalItemStateManager.getItemState(LocalItemStateManager.java:175)
        at org.apache.jackrabbit.core.state.XAItemStateManager.getItemState(XAItemStateManager.java:260)
        at org.apache.jackrabbit.core.state.SessionItemStateManager.getItemState(SessionItemStateManager.java:161)
        at org.apache.jackrabbit.core.HierarchyManagerImpl.getItemState(HierarchyManagerImpl.java:152)
        at org.apache.jackrabbit.core.HierarchyManagerImpl.buildPath(HierarchyManagerImpl.java:278)
        at org.apache.jackrabbit.core.CachingHierarchyManager.buildPath(CachingHierarchyManager.java:199)
        at org.apache.jackrabbit.core.HierarchyManagerImpl.getPath(HierarchyManagerImpl.java:395)
        ... 18 more

How to Fix (If it is not cluster then ignore cluster steps):

1) Stop all slave
2) In master create a folder called install under crx-quickstart
3) Copy files for gogo shell in install folder (From Here)
4) Use command telnet localhost 6666 from instance or from felix console go to gogo tab


5) Use command g! "login admin <admin password>"
6) Use command "rm [<UUID of Node you are having Problem in above stack trace it is 795cff24-7fd7-3f5f-a9b1-4724c3a6ff4d>]" then use command "save"
7) Then go to crx-quickstart/repository/workspace/crx.default
8) Create a file called redo_X.log where X is same as latest index_X.tar. for example if latest index tar file name is index_9_24.tar then create a file called redo_9_24.log
9) Add following entry in redo log (Based on above stack trace, This is to recreate search Index)
10) 1 DEL 795cff24-7fd7-3f5f-a9b1-4724c3a6ff4d
      1 DEL 34228aff-f648-429a-a097-29de8568fe3e
      1 COM
11) Restart master instance
12) Start slave instance

Note: Before you delete any node, You can do stat on that node to see what node is causing problem. You can do something like,

g! stat 795cff24-7fd7-3f5f-a9b1-4724c3a6ff4d
Path:         {} {}34228aff-f648-429a-a097-29de8568fe3e
NodeId:       795cff24-7fd7-3f5f-a9b1-4724c3a6ff4d    
ParentId:     34228aff-f648-429a-a097-29de8568fe3e    
NodeTypeName: {internal}Group                        
Memory used:  2400                                    
ChildNode Entries:
+ {internal}policy[1] -> c577627a-bb22-4c82-95af-6acc1881302a

Above output shows node is pointing to orphan node, Which can not be fixed by consistency check and fix.

Use help command to find things that you can do with gogo shell.

Caution: gogo shell (Similar to crx console in earlier version) is very powerful tool and cause system damage as well. Please use this tool with care.

Let me know if you want detail about any command in gogo shell.

Tuesday, October 23, 2012

How to manage Hot backup Or Manage Disaster Recovery in CQ

There are various approach, I am listing them with pros and cons


Approach 1: Use clustering, but direct the requests only to one node. In case of problems with this node just switch over to the other node. Essentially this is an active-passive scenario.
+ proven technology, documented and fully supported feature
+ automatic fail over easily possible
- additional license cost
- Latency issue might effect performance
- Managing cluster is sometime difficult

Approach 2: Use replication (on modification, no version, no status update).
+ proven technology, documented and fully supported
+ easy handling, reconfiguration during runtime
- manual reconfiguration of replication in case of switch
- "on modification" only works for cq:pages and not for other type like DAM
workflows, Events, Replication queues. (But if you activate DAM asset it will be there is stand by system)

So, this approach 2 doesn't achieve a "full-standby" system, but it more looks like a way not to loose content. Everything else is probably gone, so it's only for a really worst-case scenario.


Approach 3: Build periodically content packages and replicate them to the standby-system
+ you can also do a provisioning of other staging systems with this content
+ load on the active system predictable, normal editing actions are not
loaded with this
- self-written, not supported
- you can package workflows and events when grabbing them from repository
- Need testing to see if it will work.

Some Questions:

Q: What is best way to create active passive clustering node and what should be I careful about.
A: Create multi node cluster and make sure that DR nodes are not taking any request. You have to careful about that there is not a lot of latency between DR node active nodes.
Also it is very difficult to keep one node from active node as master in case master goes down. There is no harm of having DR as master node but not recommended (As write always goes through master). You can use either felix console or preferredMaster property to set up master in advance. Please read http://crxcluster.wemblog.com very carefully to understand CRX better.

To make a node master:
http://HOST:PORT/system/console/jmx/com.adobe.granite%3Atype%3DRepository




Since this is exposed as a JMX you can monitor it and invoke it during run time.

Q: Do I have to take cold backup of all the nodes
A: No, if you are using clustering then taking backup of any node (usually master) is enough.

Q: How about publish instance 
A: It is not recommended to use clustering in publish instance, Unless there is no other way to support some use case. In that case each publish instane is Hotback of each other. Note that you can not recover publish instance from nightly backup (As things might have been updated from the time back up was created). Usually it is recommended to have a backup publish instance which does not take load but configured as replication agent in author.

Q: I am just left with nightly backup, How should I create new publish instance ?
A: In this case you have to find out things replicated from the time last nightly backup was created and now. You can use nightly backup in conjunction with http://www.wemblog.com/2011/10/how-to-find-all-pages-modified-or.html  to support this use case.

How to work with Configurations in CQ

Use Case: You want to create configuration which can be edited at run time using OSGI console.

Background:

http://www.wemblog.com/2012/01/how-to-set-up-run-mode-in-cq-wem.html

CQ already comes with predefined configuration that you can find using following Xpath queries
//element(*,sling:OsgiConfig)

Or using query builder predicate (http://HOST:PORT/libs/cq/search/content/querydebug.html)

type=nt:file
nodename=*.config
orderby=@jcr:content/jcr:lastModified
orderby.sort=desc

From front end you can find all configurations under config tab in felix console



You can also download all configurations using felix console



You can also go to configuration through Component tab



On file systems you can find configuration under /crx-quickstart/launchpad/config folder.

Configuration can be at any of above location but order of precedence of selecting a configuration is,

1) /apps
2) /libs
3) File system

For more information please see http://dev.day.com/content/kb/home/cq5/CQ5SystemAdministration/LoadingOrderFelixConfig.html

How to Create a felix Configuration:

1) By creating a sling:OsgiConfig node using CRXDE

Step 1:

Step 2:



2) Using xml



3) By Code

How to Read Property:

1) Using generic class (You can create interface)

2) Within bundle




Cool part about configuration that I like most is, You can create environment specific configurations and based on your environment particular configuration will get selected. You can set configurations as given in http://www.wemblog.com/2012/01/how-to-set-up-run-mode-in-cq-wem.html


























Some Questions:

Q: Where does my configuration get stored when I change it through felix console directly

Ans: In CQ5.5 it get stored in repository under /apps/system/config thus get replicated across all cluster. But in CQ5.4 and before it get stored in file system under /crx-quickstart/launchpad/config/<Name of config>.config thus requires changes in all cluster instance.





















Q: What if I just want configuration for particular environment

Ans: In that case you have to set up your config for that environment and then create configuration under that node. See example above.

Q: What is recommended way to create configuration

Ans: It is recommended to change configuration through repository and not through felix console if possible. That way you can track configuration in SVN as well.

Q: I see that there is already a configuration under felix console, How can I create one under repository to override that.
Ans: To do that first go to felix console -> configuration -> open configuration and look for pid

Then create exactly same node under /apps/config<Any enviornment> of type osgiconfig


Q. How to create configuration that has drop down of selected value
A. You can do something like,

How to create Configuration factory to locate services

Some time we want to create configuration factories to create factory of configuration (For example Logger configuration)



In order to create such configuration you could use annotation

 
To read such configuration though supporting class you can use,

@References({
  @Reference (cardinality=ReferenceCardinality.OPTIONAL_MULTIPLE,        policy=ReferencePolicy.DYNAMIC, referenceInterface = MyCustomService.class, name="MyCustomService")
})


How to locate configurations in configuration factory (Note that you might not need to do this if you are using independent configuration factories, for example configuration factories of scheduler) 




Please ask any other question that you want me to add here.

Tuesday, October 2, 2012

How to implement CQ scheduler to run on a event

Use Case:

1) You want to implement a scheduler to run on a specific event. That event could be any thing ranging from any Node action (Add,Modify,Delete) or any replication action (Activate, Deactivate, Reverse)

Pre requisite:

1) http://sling.apache.org/site/eventing-and-jobs.html
2) http://felix.apache.org/site/apache-felix-event-admin.html
3) http://www.osgi.org/javadoc/r4v41/org/osgi/service/event/package-summary.html
4) http://sling.apache.org/apidocs/sling5/org/apache/sling/event/package-summary.html

Implementation:

Step 1




Step 2: Handler to perform action




Note: you can find some more code example at http://wemcode.wemblog.com

Tuesday, September 11, 2012

How to put DAM asset back to workflow for processing in CQ

Use Case: Something went wrong during Asset upload and not all steps in Update Asset workflow get executed successfully.

Impact: Some time when Assets are not uploaded properly or something went wrong during upload process, not all steps in update asset workflow completes, as a result of which some time metadata or rendition is missing. You can probably run those script again to complete process but easiest way would be to put those asset back to workflow.

Solution:

You can use following script to do this


# !/bin/bash
# Author: Yogesh Upadhyay
# The host and port of the source server
SOURCE="HOST:PORT"
# The user credentials on the source server (username:password)
SOURCE_CRED="admin:ADMIN PASSWORD"
#Filter path
ROOT_PATH="PATH OF ASSET"
IFS=$'\n'
#If you want to run for only those asset for which property is missing
MISSING_PROPERTY=dc:format
#Query to get all Dam Asset
ALL_PATHS=$(curl -s -u $SOURCE_CRED "$SOURCE/bin/querybuilder.json?path=$ROOT_PATH&type=dam:Asset&property=jcr:content/metadata/@$MISSING_PROPERTY&property.operation=not&p.limit=-1" | tr "[" "\n" | sed -e $'s/},{/\\\n/g'  | grep path | awk -F \" '{print $4 "\n"}')
#echo "$ALL_PATHS"
for SINGLE_PATH in $ALL_PATHS
do
echo "Fixing $SINGLE_PATH"
curl -s -u $SOURCE_CRED -d "model=/etc/workflow/models/dam/update_asset/jcr:content/model&payloadType=JCR_PATH&payload=$SINGLE_PATH" $SOURCE/etc/workflow/instances
sleep 3
done

Script to change wrong mimeType and dc:format of pdf document


# !/bin/bash
# Author: upadhyay.yogesh@gmail.com
# The host and port of the source server
SOURCE="localhost:4504"
# The user credentials on the source server (username:password)
SOURCE_CRED="admin:<PASSWORD>"
#Filter path
ROOT_PATH="<ROOT PATH>"
IFS=$'\n'


#Query to get all Dam Asset with extension pdf but with default mime type
ALL_PATHS=$(curl -s -u $SOURCE_CRED "$SOURCE/bin/querybuilder.json?path=$ROOT_PATH&type=dam:Asset&nodename=*.pdf&property=jcr:content/renditions/original/jcr:content/jcr:mimeType&property.value=application/octet-stream&p.limit=-1" | tr "[" "\n" | sed -e $'s/},{/\\\n/g' | grep path | awk -F \" '{print $4 "\n"}')
echo "$ALL_PATHS"
for SINGLE_PATH in $ALL_PATHS
do
echo "Fixing $SINGLE_PATH"
curl -s -u $SOURCE_CRED "-Fjcr:mimeType=application/pdf" $SOURCE$SINGLE_PATH/jcr:content/renditions/original/jcr:content
curl -s -u $SOURCE_CRED "-Fdc:format=application/pdf" $SOURCE$SINGLE_PATH/jcr:content/metadata
sleep 2
done


You can actually check for not equals property as well, But that will require custom predicate evaluator for not.



Some more interesting script:

Change thumbnail process arg to add additional thumbnail


SOURCE="localhost:4502"
# The user credentials on the source server (username:password)
SOURCE_CRED="admin:admin"
IFS=$'\n'
#Keep previous one as is this is not append operation,
#Off course this could be empty if you dont want any thumbnail
NEW_ARGS="[140:100],[48:48],[319:319],[210,210]"
PROCESS_PATH=`curl -s -u $SOURCE_CRED "$SOURCE/bin/querybuilder.json?path=/etc/workflow/models/dam/update_asset/jcr:content/flow&type=nt:unstructured&property=PROCESS&property.value=com.day.cq.dam.core.process.CreateThumbnailProcess&p.limit=-1" | tr "[" "\n" | sed 's/},{/\n/g' | grep path | awk -F \" '{print $4 "\n"}'`
echo "$PROCESS_PATH"
curl -s -u $SOURCE_CRED "-FPROCESS_ARGS=$NEW_ARGS" $SOURCE$PROCESS_PATH

Remove all thumbnail and just keep original file


# !/bin/bash
# Author: upadhyay.yogesh@gmail.com
# The host and port of the source server
SOURCE="localhost:4502"
# The user credentials on the source server (username:password)
SOURCE_CRED="admin:admin"
IFS=$'\n'
#Filter path
ROOT_PATH="/content/dam<Your path>"
ALL_PATHS=`curl -s -u $SOURCE_CRED "$SOURCE/bin/querybuilder.json?path=$ROOT_PATH&nodename=*thumbnail*png&type=nt:file&p.limit=-1" | tr "[" "\n" | sed 's/},{/\n/g' | sed 's/ /%20/g' | grep path | awk -F \" '{print $4 "\n"}'`
echo "$ALL_PATHS"
for SINGLE_PATH in $ALL_PATHS
do
echo "Removing $SINGLE_PATH"
curl -u $SOURCE_CRED -F:operation=delete $SOURCE$SINGLE_PATH
done



Put dam asset in workflow which is created between certain date


# !/bin/bash
# Author: Yogesh Upadhyay
# The host and port of the source server
SOURCE="YOUR SERVER NAME"
# The user credentials on the source server (username:password)
SOURCE_CRED="admin:password"
#Filter path
ROOT_PATH="/content/dam"

DATE_FROM="2016-03-10"
DATE_TO="2016-03-16"
IFS=$'\n'
#If you want to run for only those asset for which property is missing
#Query to get all Dam Asset

ALL_PATHS=$(curl -s -u $SOURCE_CRED "$SOURCE/bin/querybuilder.json?path=$ROOT_PATH&type=dam:Asset&daterange.property=jcr:created&daterange.lowerBound=$DATE_FROM&daterange.upperBound=$DATE_TO&p.limit=-1" | tr "[" "\n" | sed -e $'s/},{/\\\n/g'  | grep path | awk -F \" '{print $4 "\n"}')
#echo "\n"
echo "$ALL_PATHS"
#echo "\n"
for SINGLE_PATH in $ALL_PATHS
do
echo "Fixing $SINGLE_PATH"
curl -s -u $SOURCE_CRED -d "model=/etc/workflow/models/dam/update_asset/jcr:content/model&payloadType=JCR_PATH&payload=$SINGLE_PATH" $SOURCE/etc/workflow/instances
sleep 3
done


Please check http://dev.day.com/docs/en/cq/current/workflows/wf-extending.html and http://dev.day.com/docs/en/cq/current/dam/customizing_and_extendingcq5dam/query_builder.html in case you want to achieve this through code.

Note: If you have some custom property in your Asset this might override those property. Please make sure that you have tested this script before using it.

Tuesday, August 21, 2012

How to work with Version in CQ

Use Case: 

1) You have to delete specific version from repository
2) You have to create versions using API
3) You have to remove versions
4) Reduce size of repository by removing versions

Background:

CQ uses JCR version management to maintain versions

http://jackrabbit.apache.org/jackrabbit-architecture.html

http://www.day.com/maven/jsr170/javadocs/jcr-2.0/javax/jcr/version/package-summary.html

VersionManger http://www.day.com/maven/jsr170/javadocs/jcr-2.0/javax/jcr/version/VersionManager.html is main service to handle most of task related to versioning.

Within CQ in file systems versions are persisted under /crx-quickstart/repository/version and version index under /crx-quickstart/repository/repository/index

Corresponding configuration is under /crx-quickstart/repository/repository.xml (Which is global configuration)

<Versioning rootPath="${rep.home}/version">
<......>
</Versioning>

Within repository versions are stored under /jcr:system/jcr:versionStorage node, Which is one per repository.

How Versions are created in CQ:

Following actions are responsible for creating version

1) Activation of page (This is controlled by version manager configuration)
http://dev.day.com/docs/en/cq/current/deploying/configuring_cq.html#OSGi%20Configuration%20in%20the%20Repository

Note that from CQ5.5 this configuration is not available by default. You have to create a osgi:Config within repository for this (com.day.cq.wcm.core.impl.VersionManagerImpl).

You can control number of version created by activation by setting versionmanager.maxNumberVersions property.





















2) By sidekick


3) Using API http://www.day.com/specs/jcr/2.0/15_Versioning.html and example http://wiki.apache.org/jackrabbit/ExamplesPage#Versioning_Basics


How to find specific version in CQ

1) Using repository
  • Note that not for all node versions are created. Node has to be of mixin type mix:versionable (node.addMixin("mix:versionable")  
  • Once you are sure that node is of correct mixin type, Find that node using CRX explorer or CRXDE light. 
  • Once node is found find UUID of that node

  • Based on UUID you can find out location of version within repository. Within repository versions are stored like this/jcr:system/jcr:versionStorage/<FIRST TWO LETTER OF UUID>/<NEXT TWO LETTER>/<NEXT TWO LETTER>/<UUID>/<VERSION NUMBER> For example for UUID ce472450-f567-459e-8adc-6fd7a3d8a4f3 you can find it's version under /jcr:system/jcr:versionStorage/ce/47/24/ce472450-f567-459e-8adc-6fd7a3d8a4f3/<Version Number>













2) using API

Something like

VersionManager mgr = session.getWorkspace().getVersionManager();
VersionHistory vh = (VersionHistory) node.getParent().getParent();
VersionIterator vit = vh.getAllVersions();
while (vit.hasNext()) {
Version v = vit.nextVersion();
}

How to Purge Version in CQ

1) Using configuration

http://dev.day.com/docs/en/cq/current/deploying/configuring_cq.html#Configuring%20Version%20Purge

2) Using version Purge Tool

From CQ5.5 onward version purge tool is integrated in CQ that you can use to purge versions

for that go to tools -> versioning -> PurgeVersion
















Purging version will help you to reduce size of repository a lot. You can also configure your repository for not to create versions for other environments (Like dev or staging) 

Monday, July 16, 2012

How to Clean Up Nodes in CQ / WEM

Use Case:

1) Clean your repository.
2) remove unwanted nodes.

Solution: You can use following package to clean up nodes based on certain criteria.

1) Install this package using package manager






2) Go to http://HOST:PORT/apps/tools/components/cleanNode/run.html

3) Select your options

4) Select run



Note: Please use this utility with care, As it leads to deletion of node from repository. Please DO NOT use it in production without proper testing.

Thursday, July 12, 2012

How to clear replication queue in CQ / WEM

Use Case

Case 1: Your replication queue is blocking and you have no idea why it is not getting processed.
Case 2: Some one accidentally activated thousands of pages and you want to clean them now.
Case 3: You want safe way to remove replication jobs and then activate them again if required
Case 4: You want to clean a dummy or problematic replication agent

Pre requisite : CQ5.4 or CQ5.3 with replication fix pack.

Solution :

1) Install attached package using package manager

2) Go to http://<host>:<port>/apps/tools/components/cleanReplication/run.html Or go to http://<host>:<port>/etc/replication/agents.author.html click on your agent and you will will have options.

3) First do dry run and see if entries are getting deleted

4) Then check all checkbox and click on run clean replication.

5) There is also a failed replication Listener configuration, You can configure it to send an email in case replication fails.

Here are some screen shot




















Here is package:

For CQ5.4:



Q: What will go to failed queue
A: Replication Job failed even after max retry.

Q: Why not use clear replication from replication page to remove entry ?
A: Some time you won't see entry to remove. And some time you have a lot of items in queue that is not easy to remove using UI. Some time you want to delete just specific entry and they are a lot (In that case you can use filter). See use case for more detail.

Note: Since the way replication data is stored in repository changes a lot over time this might not work with earlier version of CQ till CQ5.3. You can also remove unwanted code and change it according to your Need. If there is some problem let me know and given time I can look in to it. 

Special thanks to Henry Saginor from Adobe for providing code for failed replication listener. 



How to Use Dispatcher with Mapped content

Use Case: You are using /etc/map or resource resolution to map your content and dispatcher flush is not working any more.
More information about Sling mapping can be found here http://sling.apache.org/site/mappings-for-resource-resolution.html

I have seen a lot of customer using sling mapping or resource resolver setting to map /content/<There site> to / to shorten URL. But as soon as they do that, They claim that dispatcher flush is not working. And here is reason why,

1) Dispatcher flush does not take mapping or resource resolver rule in to account to flush cache or invalidate static file.

To understand it better, Here is use case,

1) From example your site URL is somedomain.com/content/mysite/en/us/survey.html and then you shorten it to somedomain.com/en/us/survey.html by mapping rules or resource resolver rule.

2) Suppose your document root is /docroot/htdocs

3) Now when some one try to access page somedomain.com/en/us/survey.html your page will get cached under /docroot/htdocs/en/us/survey.html

4) But when you will activate this page from author page under somedomain.com/content/mysite/en/us/survey.html will get flush, As dispatcher flush agent has no idea about mapping rules.

Resolution:

Now in order to avoid this problem, You should have mod_rewrite rule along with resource resolver and mapping rules.

Here is basic example of rules you want in your mod_rewrite (These rules will differ from site to site)

RewriteRule ^/$ <Your home page>

RewriteCond %{REQUEST_URI} !^/apps/(.*) [NC]
RewriteCond %{REQUEST_URI} !^/etc(.*) [NC]
RewriteCond %{REQUEST_URI} !^/libs(.*) [NC]
RewriteCond %{REQUEST_URI} !^/content(.*) [NC]
RewriteCond %{REQUEST_URI} !^/system(.*) [NC]
RewriteCond %{REQUEST_URI} !^/dam(.*) [NC]
RewriteRule ^/(.*) /content/mysite/$1 [PT]
RewriteRule ^/(.*)\?(.*) /content/mysite/$1 [PT]

Above rule make sure that content at right path is getting flushed.

Some more trick (But will not work 100% see http://forums.adobe.com/thread/1082213 for detail):

You can also do something like

LoadModule headers_module modules/mod_headers.so
RequestHeader edit CQ-Handle /content/mysite(/.*) $1 early

Another solution at replication agent level By David to solve this issue,

http://adobe-consulting-services.github.io/acs-aem-commons/features/dispatcher-flush-rules.html

code for above is found here https://github.com/Adobe-Consulting-Services/acs-aem-commons/tree/master/bundle/src/main/java/com/adobe/acs/commons/replication

Q: Why I will use etc/map or resource resolution if these rules are enough for mapping ?
A: Mod_rewriter rules will take care of just mapping and not link rewriting. You need /etc/map or resource resolver rules for link rewriting.

Q: Why dispatcher flush do not take mapping rules in to account ?
A: There is already an enhancement request for this.

Q: I don't want to write these rewrite rules, How else I can handle this ?
A: One option is set your stat file level to 0 (Which is default), This might invalidate all resource under invalidation tag of dispatcher.any upon activation. But again if you watch your dispatcher log carefully, wrong file will get evicted on activation. But you still see updated content due to invalidation.

Q: How about vanity URL ?
A: Well thats problematic. You might want to stick with stat file or write mod_rewrite rules for them as well. 

On side note, understanding /statfileslevel is very important to improve performance of your website.

Thursday, June 14, 2012

How to Rotate Logs in CQ / WEM

Use Case:

  • Rotate all logs in CQ
  • Logs are taking a lot of space in file system
Solution:

Till CQ5.4

There are different kind of logs in CQ. Please refer http://www.wemblog.com/2012/01/how-to-change-different-log-locations.html for that

Here is process to rotate all logs,

CQ Logs:

crx-quickstart/logs/error.log :

You can rotate error log using sling:OsgiConfig under /libs/sling/config/org.apache.sling.commons.log.LogManager. Please override this under /apps/sling/config/org.apache.sling.commons.log.LogManager. You can additionally have environment specific configuration as config.author or config.publish










crx-quickstart/logs/request.log and crx-quickstart/logs/access.log:


CRX Log:

crx-quickstart/logs/crx/error.log

You can rotate CRX log through configuration in /crx-quickstart/server/runtime/0/_crx/WEB-INF/log4j.xml by changing configuration of 

<appender name="error" class="org.apache.log4j.RollingFileAppender">
        <param name="File" value="crx-quickstart/logs/crx/error.log"/>
        <param name="maxFileSize" value="10MB"/>
        <param name="maxBackupIndex" value="20"/>
        <layout class="org.apache.log4j.PatternLayout">
            <param name="ConversionPattern" value="%d{dd.MM.yyyy HH:mm:ss} *%-5p* %c{1}: %m (%F, line %L)%n"/>
        </layout>
    </appender>

/crx-quickstart/logs/{stdout,stderr}.log

These logs are not controlled by sling or log4j. You have to use apache log rotate utilities to do this. You might have to do following to rotate these logs (By default they are not rotated)

After
export CQ_LOGDIR
CQ_LOG="${CQ_LOG:-"$CQ_LOGDIR/startup.log"}"

------- ADD ------

CRX_LOG_STDOUT="${CQ_LOG:-"$CQ_LOGDIR/../../logs/stdout.log"}"
export CRX_LOG_STDOUT

CRX_LOG_STDERR="${CQ_LOG:-"$CQ_LOGDIR/../../logs/stderr.log"}"
export CRX_LOG_STDERR

And change your rotate option as,

exec $jvmExe 2>&1 | /usr/sbin/rotatelogs "$CQ_LOG.%Y%m%d" 86400
| /usr/sbin/rotatelogs "$CRX_LOG_STDOUT.%Y%m%d" 86400 | /usr/sbin/rotatelogs "$CRX_LOG_STDERR.%Y%m%d" 86400

That mean rotate both log after 86400 second (1 day).


Server Logs

crx-quickstart/server/logs/{access,startup}.log

Server logs can not be rotated at sling level. You need to rotate these logs at OS level using logrotate utility similar to http://httpd.apache.org/docs/2.0/programs/rotatelogs.html

For this in serverctl script you can do following (Adjust log file location accordingly),

After
export CQ_LOGDIR
CQ_LOG="${CQ_LOG:-"$CQ_LOGDIR/startup.log"}"

------- ADD ------

CQ_LOG_ACCESS="${CQ_LOG:-"$CQ_LOGDIR/access.log"}"
export CQ_LOG_ACCESS

And change your rotate option as,

exec $jvmExe 2>&1 | /usr/sbin/rotatelogs "$CQ_LOG.%Y%m%d" 86400
| /usr/sbin/rotatelogs "$CQ_LOG_ACCESS.%Y%m%d" 86400

That mean rotate both log after 86400 second (1 day).

If you do not have logrotate utility, OOTB CQ support rotation based on size (Not on date or number), For that go to /crx-quickstart/server/etc/server.xml and add following line

<log-file>
<log-file-name>logs/access.log</log-file-name>
<log-max-file-size>20MB</log-max-file-size>
</log-file>

<log-file>
 <log-file-name>../logs/server.log</log-file-name>
 <log-max-file-size>20MB</log-max-file-size>
</log-file>

Launchpad Logs

crx-quickstart/launchpad/logs/error.log

You can configure launchpad error log using felix configuration or creating a custom configuration in crxde may be under /apps/sling/config. Please see http://sling.apache.org/site/logging.html for more information. You can also control sling logs rotation though /crx-quickstart/launchad/sling.properties

Look for, following properties
org.apache.sling.commons.log.file.number=5
org.apache.sling.commons.log.file.size='.'yyyy-MM-dd
org.apache.sling.commons.log.file=${sling.home}/logs/error.log




Dispatcher log:

For dispatcher log you have to use http://httpd.apache.org/docs/2.0/programs/rotatelogs.html (Similar to server log). Something like

DispatcherLog '|/usr/local/apache/bin/rotatelogs <Path-to-your-log>/logs/%Y-%m-%d-dispatcher.log 86400'

http://dev.day.com/content/kb/home/Dispatcher/faq-s/SetupDispatcherLogRotation.html


CQ5.5

Good thing about CQ5.5 is there is not different locations to control log rotation. You can control all log rotations through sling configuration http://sling.apache.org/site/logging.html. One thing to note here is, unlike CQ5.4 ../logs/<your-log> will point to logs folder under CQ root and not under /crx-quickstart/logs. If you want your logs to go under /crx-quickstart/logs you have to use log location as logs/<your-log> in sling configuration.

It is always recommended to modify your log configuration under /apps/<Path>, That way you can port it to different environment.

Some more reference 



Note: For OS level log rotation, You can use any log rotation you want and give path to CQ log files.

You can try this option as well, If log rotate utility option is not working after config through serverctl.

$NOHUP $jvmExe | /usr/sbin/rotatelogs "$CQ_LOG.%Y%m%d" 86400>> /dev/null 2>&1

Thursday, June 7, 2012

How to add a custom Login Module in CQ / AEM

Use Case:

1) You want to create a custom login module in CQ / AEM
2) You already have a custom login module in CQ5.4 and integrate it with CQ5.5

Prerequisite: Please check http://dev.day.com/content/kb/home/cq5/CQ5Troubleshooting/cq55prerelease-installandconfigchanges.html

Solution:

CQ 5.6 and less

Any custom login module should be added in CQ5.6 and less as a JAAS module. That mean CQ should have access to module during load time.

Here is set of steps you might have to do to create custom login module

1) Create a OSGI bundle with class that extend Jackrabbit AbstarctLoginModule (Example http://wemcode.wemblog.com/custom-login-module)

2) Create bundle as frangment bundle attached it to com.day.crx.sling.server



<plugin>
<groupId>org.apache.felix</groupId>
<artifactId>maven-bundle-plugin</artifactId>
<extensions>true</extensions>
<configuration>
<instructions>
<Import-Package>!com.day.crx.core.token,!org.apache.jackrabbit.*,*</Import-Package>
<Bundle-SymbolicName>${project.artifactId}</Bundle-SymbolicName>
<Export-Package>....... Package you want to export .... </Export-Package>
<Fragment-Host>com.day.crx.sling.server</Fragment-Host>
<Bundle-Name>${project.name}</Bundle-Name>
<Bundle-Description>${project.description}</Bundle-Description>
<Bundle-Version>${project.version}</Bundle-Version>
<Include-Resource>lib=${basedir}/lib</Include-Resource>
<Bundle-ClassPath>
.,
lib/<Your custom jar file>.jar,
</Bundle-ClassPath>
</instructions>
</configuration>
</plugin>

3) Deploy and start your bundle as start level 15. You can do following things for that

curl -u admin:admin -T <Your custom module bundle>.jar http://<host>:<port>/apps/<Your path>/install/15/<Your login module>.jar

Or


<profiles>
<profile>
<id>install-packages</id>
<build>
<plugins>
<plugin>
<groupId>org.apache.sling</groupId>
<artifactId>maven-sling-plugin</artifactId>
<executions>
<execution>
<id>install-package</id>
<goals>
<goal>install</goal>
</goals>
<configuration>
<slingUrl>${crx.url}/system/console/install</slingUrl>
<user>${crx.user}</user>
<password>${crx.password}</password>
<bundleStartLevel>15</bundleStartLevel>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>

4) Change repository.xml 

Open the file crx-quickstart/repository/repository.xml (on the server's filesystem) and remove the <LoginModule>...</LoginModule> element.
Find the <SecurityManager> element and add the WorkspaceAccessManager as shown below:
<SecurityManager class="com.day.crx.core.CRXSecurityManager">
<WorkspaceAccessManager class="org.apache.jackrabbit.core.security.simple.SimpleWorkspaceAccessManager"/>
<UserManager class="com.day.crx.core.CRXUserManagerImpl">
<param name="usersPath" value="/home/users"/>
<param name="groupsPath" value="/home/groups"/>
<param name="defaultDepth" value="1"/>
</UserManager>
</SecurityManager>

5) Create custom jaas config

Now create the file crx-quickstart/conf/jaas.conf. In this file you would have something like this:

com.day.crx {
<Your login Module class name> sufficient
com.day.crx.core.CRXLoginModule sufficient;
};

6) Change start up script

Update your crx-quickstart/bin/start script with this jvm parameter (Replace /path/to/your/jaas.conf with the real path to the jaas.conf file):
-Djava.security.auth.login.config=/path/to/your/jaas.conf

7) Restart your CQ

Note: if you already have Non osgi version of your custom login module, You can use http://www.wemblog.com/2012/04/how-to-integrate-3rd-party-jar-file-in.html to convert to OSGI bundle. Make sure that you follow step 2.

To debug custom login module, you can follow steps mention in http://www.wemblog.com/2011/09/how-to-set-up-debug-mode-for.html

Note that LDAP login module com.day.crx.security.ldap.LDAPLoginModule in CQ is good example of custom Login Module.

AEM / CQ 6

From CQ6 onward, Login module can now be configured as pluggable login module in OSGI. To implement Login Module in CQ6 onward you can

Option 1:

implement org.apache.sling.jcr.jackrabbit.server.security.LoginModulePlugin Here is example of Pluggable Login Module with Authentication handler http://svn.apache.org/repos/asf/sling/trunk/bundles/auth/form/src/main/java/org/apache/sling/auth/form/impl/

More info: https://sling.apache.org/documentation/the-sling-engine/authentication/authentication-authenticationhandler/form-based-authenticationhandler.html

Option 2:

You can also use OAK external Login Module https://github.com/apache/jackrabbit-oak/tree/trunk/oak-auth-external To create custom login module (Which you register and service) as well. Example of that would be LDAP Login Module in AEM6 https://github.com/apache/jackrabbit-oak/tree/trunk/oak-auth-ldap

Here is one more example of how to use this in AEM 6 onward: https://helpx.adobe.com/experience-manager/using/oak-login.html


You do not have to restart your instance after creating custom login module in AEM 6.

Saturday, June 2, 2012

How to add a new supported language in CQ / WEM

Use case: 

  • You want to add new language to CQ
  • Change display language options in translator grid
  • Change language name and default countries 

Solution:

You can access translator UI in CQ with following URL http://<HOST>:<PORT>/libs/cq/i18n/translator.html



Create new language location for Dictionary

Go to CRXDE lite (or your favorite JCR browser) and add this structure (assuming /apps/myproject/i18n as a typical location for custom apps):
/apps/myproject/i18n [sling:Folder]
    - de [nt:unstructured] [mix:language]
        + jcr:language = de
    - fr [nt:unstructured] [mix:language]
        + jcr:language = fr
Then reload the translator and the path /apps/myproject/i18n should show up in the drop-down at the top.
Note: the translator will only save translations for languages that are actually present underneath the path (e.g. /apps/myapp/i18n), others will be skipped.

Change List of Languages Display in Grid

To change what languages are shown in the translator's grid, create a multi-value string property /etc/languages/languages (node structure must be created), containing the iso language codes (e.g. ["de", "fr", …]). The default is hard coded in the translator and maps the ootb CQ languages: ['de', 'fr', 'it', 'es', 'ja', 'zh-cn']


Add supported language in WCM

  • You need to overlay/overwrite /libs/cq/security/widgets/source/widgets/security/Preferences.js to change the list of available languages (used in security admin)
  • and overlay /libs/cq/security/content/tools/userProperties (inside change language list under items/common/items/lang/options - similar dialog to Preferences.js) (used for user drop-down in the top right on admin consoles)
  • Then you need to add the new language somewhere in the repo, e.g. /apps/yourapp/i18n, following the layout as under /libs/wcm/core/i18n (see also the above section on the translator UI). Stuff under /libs must not be changed!
Here is a sample package:
  • adds /apps/test/i18n with translation of "New..." in zh-cn ("simplified") and zh-tw ("traditional") languages
  • overlays /apps/cq/security/content/tools/userProperties and /apps/cq/security/widgets/source/widgets/security/Preferences.js to add the zh-tw language
  • to check, set the current user's language to "Traditional Chinese" and look in the siteadmin: the first button in the toolbar should now be "traditional" instead of "New..."
change language name and default country 

Note: since 5.5 load 21
This means:
  • language titles
  • country names
  • default countries (for codes like "en", "de", etc.)
A language list is stored under /libs/wcm/core/resources/languages. It can be overlayed by copying it to /apps/wcm/core/resources/languages and changing or extending the list there. If you do so, also change the configuration of the language manager com.day.cq.wcm.core.impl.LanguageManagerImpl: set langmgr.list.path = /apps/wcm/core/resources/languages at 
http://localhost:4502/system/console/configMgr/com.day.cq.wcm.core.impl.LanguageManagerImpl.

The default countries, which come into play when resolving the country for a language code such as "en" ("en_GB" would be such a default country), and in turn for displaying flags in the UI, are configured in this list as well. A property defaultCountry on a language node must contain the full code, such as "ja_jp", which would define "jp" as default country for "ja".

Information is provided by Alexander Klimetschek  from Adobe. Many thanks to him. 


Wednesday, May 2, 2012

How to use dispatcher to serve cache content if render is unavailable in CQ / WEM

Use Case: You want to serve cached content if all the renderer in dispatcher.any is unavailable

Prerequisite: You need dispatcher version 4.1.0. You can download latest dispatcher from package share. You can also connect to Day Care to get latest dispatcher release.

Problem: Currently If all renderer is unavailable, You get "502 Bad response code". It might possible that all your pages are already cached before render went down. It will make sense to serve cached content in that case.

Solution: You can use following parameter in dispatcher.any for this,

/ServeStaleOnError "1"


You have to restart apache after making changes. You can add this tag any where at /website level. So it should be,


/farms
  {
  # first farm entry (label is not important, just for your convenience)
  /website
    {
....
/ServeStaleOnError "1"
}
}

If you want to serve custom error page if pages are not cached. Feel free to modify httpd.conf and add following line to it,

ErrorDocument 502 <Path for custom 502 page>

This is true for any error page. More information can be found here http://httpd.apache.org/docs/2.0/mod/core.html#errordocument

If you are using this make sure that DispatcherPassError is set to 0. More information about this can be found here, 

You can also handle 5XX error effectively by adjusting following params in dispatcher.any
    /health_check
      {
      # Page gets contacted when an instance returns a 500
      }
    /retryDelay "1"
    /numberOfRetries "5"
    /unavailablePenalty "1"
Again You can find more information about it on Day Doc. 

Another cool thing you can do with latest dispatcher is, You can use * for client header section, So that you don't have to add all entry. Something like

 /clientheaders {*}

Feel free to comment if something is not clear.

How to Associate Digital Right Management (DRM) with a asset in CQ5.5

Use Case: You want to to associate licensing information to an Asset in DAM before serving it to user.

Solution: With CQ5.5, Now you can associate Licensing information to any Asset before serving this to user.

To do this, You have to add additional metadata to Asset called xmpRights:WebStatement that will internally point to any resource that you want to show before actual Image. CQ OOTB has one example under /content/dam/geometrixx/drm








You can also use Adobe Drive Integration with PS or Bridge to make these changes as well. Information about Adobe drive and Bridge can be found here
[1] http://www.adobe.com/products/adobedrive.html
[2] http://www.adobe.com/products/bridge.html











You can also modify asset editor page to add external metadata as well.


First, we need to create the necessary folder structure:

curl -u admin:admin -Fjcr:primaryType=sling:Folder http://localhost:4502/apps/dam

curl -u admin:admin -Fjcr:primaryType=sling:Folder http://localhost:4502/apps/dam/content/asseteditors/application/pdf

curl -u admin:admin -Fjcr:primaryType=sling:Folder http://localhost:4502/apps/dam/content/asseteditors/image/jpeg

curl -u admin:admin -Fjcr:primaryType=sling:Folder http://localhost:4502/apps/dam/content/asseteditors/image/tiff


Then we can copy the nodes from libs:

curl -u admin:admin -F:operation=copy -F:dest=/apps/dam/content/asseteditors/ http://localhost:4502/libs/dam/content/asseteditors/formitems

curl -u admin:admin -F:operation=copy -F:dest=/apps/dam/content/asseteditors/application/pdf/ http://localhost:4502/libs/dam/content/asseteditors/application/pdf/formitems

curl -u admin:admin -F:operation=copy -F:dest=/apps/dam/content/asseteditors/image/ http://localhost:4502/libs/dam/content/asseteditors/image/formitems

curl -u admin:admin -F:operation=copy -F:dest=/apps/dam/content/asseteditors/image/jpeg/ http://localhost:4502/libs/dam/content/asseteditors/image/jpeg/formitems

curl -u admin:admin -F:operation=copy -F:dest=/apps/dam/content/asseteditors/image/tiff/ http://localhost:4502/libs/dam/content/asseteditors/image/tiff/formitems

 curl commands to create the fields for the image editor:


curl -u admin:admin "-FfieldLabel=Right Management URL" -Fjcr:primaryType=cq:Widget -Fname=./xmpRights:WebStatement -Fxtype=textfield

http://localhost:4502/libs/dam/content/asseteditors/formitems
curl -u admin:admin "-FfieldLabel=Right Management URL" -Fjcr:primaryType=cq:Widget -Fname=./xmpRights:WebStatement -Fxtype=textfield

http://localhost:4502/libs/dam/content/asseteditors/application/pdf/formitems
curl -u admin:admin "-FfieldLabel=Right Management URL" -Fjcr:primaryType=cq:Widget -Fname=./xmpRights:WebStatement -Fxtype=textfield http://localhost:4502/libs/dam/content/asseteditors/image/formitems

curl -u admin:admin "-FfieldLabel=Right Management URL" -Fjcr:primaryType=cq:Widget -Fname=./xmpRights:WebStatement -Fxtype=textfield http://localhost:4502/libs/dam/content/asseteditors/image/jpeg/formitems

curl -u admin:admin "-FfieldLabel=Right Management URL" -Fjcr:primaryType=cq:Widget -Fname=./xmpRights:WebStatement -Fxtype=textfield http://localhost:4502/libs/dam/content/asseteditors/image/tiff/formitems
















You can also create your own page under dam/components/drm/pages/license, But any location is fine as long as you have right URL.


























After correct URL you might see your License agreement file before Image is served




















You might also have to change configuration to accept license agreement. For that please do following

·         Got to http://<host>:<port>/system/console/configMgr
·         Search for “DAM Event Recorder”, click it to change the configuration
·         Click “Enable this service”.
·         Save and try accepting the license again.

After User Agree with License, They should be able to view page

Note:

This is still work in progress and might change in future releases.