Monday, November 19, 2012

How to Fix No Such Item repository corruption in CQ5.5

Use Case: You have repository corruption that can not be fixed using consistency check and fix.

Prerequisite:

http://helpx.adobe.com/crx/kb/RepositoryInconsistency.html

http://helpx.adobe.com/cq/kb/cq-5-5--rien-ne-va-plus---i-need-a-text-shell.html

Stack Trace:

05.11.2012 00:30:01.476 *ERROR* [pool-5-thread-2] Unable to perform role import javax.jcr.ItemNotFoundException: failed to build path of 795cff24-7fd7-3f5f-a9b1-4724c3a6ff4d
        at org.apache.jackrabbit.core.HierarchyManagerImpl.getPath(HierarchyManagerImpl.java:400)
        at org.apache.jackrabbit.core.CachingHierarchyManager.getPath(CachingHierarchyManager.java:233)
        at org.apache.jackrabbit.core.ItemImpl.getPrimaryPath(ItemImpl.java:188)
        at org.apache.jackrabbit.core.NodeImpl.getPrimaryPath(NodeImpl.java:2753)
        at org.apache.jackrabbit.core.ItemImpl$2.perform(ItemImpl.java:379)
        at org.apache.jackrabbit.core.ItemImpl$2.perform(ItemImpl.java:376)
        at org.apache.jackrabbit.core.session.SessionState.perform(SessionState.java:216)
        at org.apache.jackrabbit.core.ItemImpl.perform(ItemImpl.java:91)
        at org.apache.jackrabbit.core.ItemImpl.getPath(ItemImpl.java:376)
        at <Some Custom Code>
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.jackrabbit.core.state.NoSuchItemStateException: 34228aff-f648-429a-a097-29de8568fe3e
        at org.apache.jackrabbit.core.state.SharedItemStateManager.getItemState(SharedItemStateManager.java:285)
        at org.apache.jackrabbit.core.state.LocalItemStateManager.getNodeState(LocalItemStateManager.java:110)
        at org.apache.jackrabbit.core.state.LocalItemStateManager.getItemState(LocalItemStateManager.java:175)
        at org.apache.jackrabbit.core.state.XAItemStateManager.getItemState(XAItemStateManager.java:260)
        at org.apache.jackrabbit.core.state.SessionItemStateManager.getItemState(SessionItemStateManager.java:161)
        at org.apache.jackrabbit.core.HierarchyManagerImpl.getItemState(HierarchyManagerImpl.java:152)
        at org.apache.jackrabbit.core.HierarchyManagerImpl.buildPath(HierarchyManagerImpl.java:278)
        at org.apache.jackrabbit.core.CachingHierarchyManager.buildPath(CachingHierarchyManager.java:199)
        at org.apache.jackrabbit.core.HierarchyManagerImpl.getPath(HierarchyManagerImpl.java:395)
        ... 18 more

How to Fix (If it is not cluster then ignore cluster steps):

1) Stop all slave
2) In master create a folder called install under crx-quickstart
3) Copy files for gogo shell in install folder (From Here)
4) Use command telnet localhost 6666 from instance or from felix console go to gogo tab


5) Use command g! "login admin <admin password>"
6) Use command "rm [<UUID of Node you are having Problem in above stack trace it is 795cff24-7fd7-3f5f-a9b1-4724c3a6ff4d>]" then use command "save"
7) Then go to crx-quickstart/repository/workspace/crx.default
8) Create a file called redo_X.log where X is same as latest index_X.tar. for example if latest index tar file name is index_9_24.tar then create a file called redo_9_24.log
9) Add following entry in redo log (Based on above stack trace, This is to recreate search Index)
10) 1 DEL 795cff24-7fd7-3f5f-a9b1-4724c3a6ff4d
      1 DEL 34228aff-f648-429a-a097-29de8568fe3e
      1 COM
11) Restart master instance
12) Start slave instance

Note: Before you delete any node, You can do stat on that node to see what node is causing problem. You can do something like,

g! stat 795cff24-7fd7-3f5f-a9b1-4724c3a6ff4d
Path:         {} {}34228aff-f648-429a-a097-29de8568fe3e
NodeId:       795cff24-7fd7-3f5f-a9b1-4724c3a6ff4d    
ParentId:     34228aff-f648-429a-a097-29de8568fe3e    
NodeTypeName: {internal}Group                        
Memory used:  2400                                    
ChildNode Entries:
+ {internal}policy[1] -> c577627a-bb22-4c82-95af-6acc1881302a

Above output shows node is pointing to orphan node, Which can not be fixed by consistency check and fix.

Use help command to find things that you can do with gogo shell.

Caution: gogo shell (Similar to crx console in earlier version) is very powerful tool and cause system damage as well. Please use this tool with care.

Let me know if you want detail about any command in gogo shell.

9 comments:

  1. I am seeing these errors on a brand new cluster set-up by cloning the Master to create the slave. Should the above steps be followed?

    ReplyDelete
    Replies
    1. This is a plain 5.5 cluster

      Delete
    2. Hello I would suggest to create a daycare ticket to find out why this issue is coming. If you are seeing this error in master as well then you might have to first fix it in master using instruction above and then start slave.

      Yogesh

      Delete
    3. Thanks Yogesh. I did raise a ticket with Adobe and it was resolved. Recreating slave from Master clone helped resolve the issue. Mentioned your blog and the person I spoke to was full of praise for you :) Thanks for your help to CQ community.

      Delete
  2. Hi,
    I am trying to use login admin command GOGO(http://localhost:4502/system/console/gogo) but i am getting login command not found.the same is happening with crx commands

    ReplyDelete
    Replies
    1. Have you installed gogo shell first using package manager or through file system as mentioned in http://helpx.adobe.com/experience-manager/kb/cq-5-5--rien-ne-va-plus---i-need-a-text-shell.html

      Delete
    2. Hi Yogesh,
      I have installed the jars using using felix console(http://localhost:4502/system/console/bundles).I am able to use GOGO tab in the felix and also able to use help and lb commands but for all commands crx commands such as crx:help and login I am getting command not found.

      Delete
    3. Hello,
      What happen when you run this command through GOGO shell in felix console it self ? Note that your CRX should be up in order for this command to run.
      Yogesh

      Delete
  3. Hi Yogesh,
    Experiencing the following error :-
    java.lang.Thread.State: BLOCKED (on object monitor) at
    org.apache.jackrabbit.core.CachingHierarchyManager.get(CachingHierarchyManager.java:499) - waiting to lock <0x000000054b406528> (ajava.lang.Object)at
    org.apache.jackrabbit.core.CachingHierarchyManager.buildPath(CachingHierarchyManager.java:194) ......
    Can you please provide your expertise?

    ReplyDelete