Wednesday, December 10, 2014

How to Set Up Clustering In CQ/AEM 6 using MongoDB

Background:


With CQ / AEM 6 TarPM is not supported any more. AEM 6 ships with Oak which for now support TarMK and MongoMK Microkernal OOTB. More information about what is New Can be found from http://www.slideshare.net/AEMHub2014/oak-michael-marth . With this change Support from Clustering is moved to actual storage layer it self (Which make more sense, given supporting all issues for clustering in earlier version). TarMK does not have replication or sharding feature so it comes down to MongoDB which support replication and sharding and hence enable High Availability (HA through replication) and Scalability (Through Sharding, Though this is still a question ?? See note below) through clustering in CQ /AEM 6.

Here we will give step by step instruction of how to set up clustering using MongoDB in CQ

Pre requisite:


There are two cases for setting Up Replica Set:

Set up a new MongoDB Instance:

  • Set up additional MongoDB instance based on instruction above
  • Start any one of instance using ./mongod --port <Your Port> --dbpath <Your DB Path> --replSet <Replica Set Name could be any thing> &  
  • You can also use configuration file to do that. More instruction here http://docs.mongodb.org/manual/tutorial/deploy-replica-set/
  • Once Mongo DB is started you can add additional replica using following instruction 
  • Once Replica set is up, Now set Up AEM
  • Then You can go to each Mongo Instance and check of data is coming using Mongo Log
Convert Existing Mongo Instance:

  • Stop you AEM instance
  • Use Following instruction to convert Mongo to replica
  • Once this is set Change AEM start script to add mongo replica instance as given in approach one 
  • start your AEM instance
  • AEM should be part of replica set now

Backup and Restore

Please check https://docs.mongodb.org/v3.0/tutorial/backup-and-restore-tools/ for MongoDB instruction of backup and restore. 

Automated script can be found here: https://github.com/micahwedemeyer/automongobackup/blob/master/src/automongobackup.sh just put this script under /etc/cron.daily and you are set for backup.

Some Common Questions

Should I set up my AEM author instance on MongoDB

Unless you have clustering requirement, I would not suggest to set up your author instance with MongoDB. Mainly because of administrative overhead.

Should I set up my AEM publish instance on MongoDB

Same as above, Unless you have a requirement which requires shared content generation I would suggest not to use MongoDB. With AEM communities, now you have an option to add Mongo Persistence for community feature at any time. Here is more detail https://docs.adobe.com/docs/en/aem/6-1/administer/communities/srp/msrp.html and https://docs.adobe.com/docs/en/aem/6-1/administer/communities/srp/msrp/demo-mongo.html

Should I store Blobs in MongoDB as well in AEM

It is not recommended to store Blob data with MongoDB. There are other options like, Local Storage, NAS, AWS you can use in that case. More detail https://docs.adobe.com/content/docs/en/aem/6-1/deploy/platform/aem-with-mongodb.html#AEM Configuration and https://docs.adobe.com/docs/en/aem/6-1/deploy/platform/data-store-config.html

How can I secure my MongoDB deployment with AEM

Notes:


1) Mongo Replication Only Provide High Availability (HA) it does not provide scalability. For scalability you need to use Sharding feature provided by Mongo. However I am not sure what would be best key to create shard on for Mongo. You can create Shard based on _id attribute. More information about sharding can be obtained here http://docs.mongodb.org/manual/sharding/  . If you are using Sharding I would suggest to use sharding with replication (Shard and then replicate shard instance) to provide both HA and scalability.

2)  There are many feature available in Mongo Replication where you can make certain replica instance read only (Data Center replica), you can use this to avoid high latency across Data Center here is all configuration you can do on Mongo http://docs.mongodb.org/manual/administration/replica-set-member-configuration/

3) MongoDB recently released MMS https://mms.mongodb.com/ to monitor and deploy Mongo Cluster easily. This will be useful if you are worried about administrative cost for Mongo 

4) If you don't want to store large documents in Mongo feel free to use custom Data Store using instruction here http://jackrabbit.apache.org/oak/docs/osgi_config.html

5) Mongo Recently launched another feature of pluggable datastore. You can use this for faster read and write based on your requirement (For example Primary with high Write Enabled Storage Like SSD or something and read with cheap storage). More info here https://www.mongosoup.de/blog-entry/A-closer-look-at-pluggable-storage.html (Official Doc yet to come)

6) Official AEM Documentation: https://docs.adobe.com/content/docs/en/aem/6-1/deploy/platform/aem-with-mongodb.html


Finally .... Some more Mongo Command ...



Special Thanks To Nelson Mei for Setting up POC for Mongo with AEM

54 comments:

  1. This looks like an awesome article. I have to start the conversion of TarMK to MongoMK however found the migration tool doesn't work so going to use VLT instead. (saw reported bug in Jira)

    Thanks for your hardwork I will be stepping through this i the coming weeks.

    ReplyDelete
    Replies
    1. Hello Eric,

      Not sure why you are using VLT, you mean you are trying to migrate everything using vlt sync ?

      You might lose version and audit info in this process I guess.

      Yogesh

      Delete
  2. Thanks for the great information and for all of the other posts I have used as resources! This site has really become an awesome source of little know configs and tidbits to point me in the right direction.

    I have a question related to a combined TarMK and MongoMK configuration shown here:
    http://docs.adobe.com/docs/en/aem/6-0/deploy/recommended-deploys.html#MongoMK%20with%20Active/Active%20Cluster

    If I am using TarMK for the Author, does the publish agent point to just one Publish Instance (since they all share a single backend ), all Publish Instances or Directly to MongoDB? If it does point to MongoDB, what does the configuration for the Publish Agent look like?

    I appreciate any direction you can provide, thanks!

    ReplyDelete
    Replies
    1. If all publish share common data, for example user generated content, that needs to be in sync in all publish at same time. You can use MongoMK in that case. In that setting all publish agent will point to common Mongo replica Cluster as shown above. You can also have a setting where master Mongos can be used for write and other for read. This will help to improve read latency in such set up.

      Yogesh

      Delete
    2. Thanks! I understand having the publish instances using MongoMK and pointing to a common MongoDB, but if I am using TarMK with an Author Instance creating Authored Content for those Publish Instances, how is the content replicated? By pointing to one of the Publishes or by pointing it to the MongoDB directly somehow?

      Delete
    3. Hello George,

      Sorry for confusion. You need to set up replication to just 1 publish instance (Publish pointing to master replica preferred) in this case. As replication happen at Mongo level for other publish instances.

      Yogesh

      Delete
    4. I appreciate the clarification! The diagram showed the Author pointing to MongoDB and I was not sure if it should be replicating directly or through a Publish Instance. It makes sense to let the one Publish ingest the replication and store them in MongoMK for all other Publishes to access. But if there was a way to Publish directly to the MongoDB from the Author instance, I thought I should try it!

      Thanks again, and your "-Doak.mongo.db=" config example saved me a bunch of time. It seems to be missing from quite a few of the other examples I have seen.

      Delete
    5. I did not see diagram. But that is possible if you are using same Mongo for both author and publish (You need to create different collection in same database however). But even in that case you need to set up at least one replication agent.

      Yogesh

      Delete
  3. Hello Yogesh, Thanks for the clear explanation on aem and mongo integration. Please find below my clarification and let me know your thoughts
    We plan to set up 1 author with 3 publish instances(publish1, publish2, publish3) . Author is planned as TarMK and the publish would be of mongodb. We have 3 mongo instances of 1 primary (mongo1) and 2 secondary (mongo2, mongo3).
    publish1 is installed with pointing to mongo1 and we were able to replicate from author to publish1 using the tree activation.
    Now when we tend to install publish2 pointing to mongo1 or the other instances of mongo2 or mongo3, we get the following errors
    ---------------------
    java.lang.RuntimeException: Error occurred while obtaining InputStream for blobId [6c7055bf3fa87c47c2ade09f80849db9e4727e07#10816]
    ---------------------------------------------------

    And my concern is whether the individual publish instances are to installed with the fresh mongo and then at a later point mongo are to be replicated.

    ReplyDelete
    Replies
    1. Hello Krish,

      Any reason why pub2 and pub3 is pointing to mongo2 and mongo3 only ? In theory you should use same mongo config across all pub. As at one point there is only one master (Primary) in a mongo cluster (Replication). Pointing publish instance to just slave might not work. For more detail please check http://docs.mongodb.org/manual/core/replication-introduction/

      Yogesh

      Delete
  4. By looking at this wonderful article, you are giving us inspiration, inspiration. The extraordinary degree Delhi Escorts that's scattered here is amazing. It helped me to refine a ton. A responsibility of gratitude is for your efforts and support, keep sharing such things for each situation. Your blog is appreciated !! Escorts in Delhi

    ReplyDelete
  5. Many thanks for making the sincere effort to explain this. I feel fairly strong about it and would like to read more. If it's OK, as you find out more in depth knowledge, would you mind writing more posts similar to this one with more information?

    suits on rent in delhi

    lehenga on rent in lajpat nagar

    lehenga on rent in delhi

    dresses on rent

    party wear lehenga on rent in delhi

    rent dresses in delhi

    clothes on rent in delhi

    party dresses on rent in delhi
    sherwani on rent in delhi
    gowns on rent in delhi

    ReplyDelete
  6. . You don’t need to worry about your monthly budget as you can enjoy each and every service in the best possible way that you dream of. Don’t let any of your sexual desires go unnoticed when you spend your quality time with our hot and sexy girls. Come and enjoy your hidden fantasies at affordable Chennai escort services rates to the fullest.

    independent escorts in chennai
    chennai independent escorts
    independent escorts in chennai
    chennai escorts services
    chandigarh escorts services
    jaipur independent escorts

    ReplyDelete
  7. Find Delhi call girls and escorts online today at missneha.com! India's #1 escort website. Over 5000 Delhi call girl adverts posted both incall & outcall.
    Delhi Escorts

    ReplyDelete
  8. Fine best Delhi Escorts service in our agency and fulfill ever sexual dream of your life.

    ReplyDelete
  9. you can see actual pics one hundred% of them and the services performed by each. on this manner you could pick the female that pleasant fits your private preferences. https://www.payalpandya.com/junnasandra-escorts.html
    bangalore escorts
    bangalore escorts services
    bangalore escorts agency

    ReplyDelete
  10. you have in mind you may do with those excessive profile in Bangalore . you may begin as you like. With a mild rubdown that opens your senses and leaves you receptive day-to-day the delight so as to come later.
    https://bit.ly/3487vuL
    bangalore escorts
    bangalore escorts services
    bangalore escorts agency

    ReplyDelete
  11. the fantasies that hang-out you for a long time, but you have not been capable of carrying out. perhaps your associate considers positive practices as taboos and also you do not dare to invite him what you would love day-to-day do
    https://bit.ly/35ko3j7
    bangalore escorts
    bangalore escorts services
    bangalore escorts agency

    ReplyDelete
  12. The site is having really good content. Keep trying to do better. Try EBookTrip to get your travel solution/ flight booking. Book Your flights to get huge discounts and unpublished flight deals for your honeymoon trip. You can call us at +1(800)404-0025 or visit us at EBookTrip.

    ReplyDelete

  13. The hectic world, a usual silhouette of wrinkled-faces and lack of time & good people make absolute absence of Love in our lives. Only a smile means to exist in the whole world bringing itself a Comet of Love. So it takes you believe on Escorts in Dehradun liable to fulfill your lives with real sense of love. Check our other Services...
    Call Girls in Dehradun
    Escorts Services in New Delhi 
    Russian Call Girls in Gurgaon
    Escorts in Rishikesh

    ReplyDelete


  14. Apart from normal escorting, we will give you memorable experience with our Escorts Service in Faridabad. worth for you money. Just tell us what you want. We will try to full fill your each fantasy. Dinner with our beauty full girls, swim, bath, travel with your partner.
    Check our other services...
    Russian Escorts in Faridabad
    Call Girls in Agra
    Escorts Service in Agra
    Escorts in Agra

    ReplyDelete
  15. I should state I motivated with your work. I needed to scrutinize such kind of article which is one of these, I've found here a lot of entrancing information. this site ideal for the understanding I need.
    escortriyadelhi
    delhi-callgirls
    delhiescortse
    sapnadelhi
    dimple-gupta
    247callgirls

    ReplyDelete
  16. We provide a whole Alternative to all of your Actual physical wants and Organic cravings also. We equilibrium your misbalanced sexual lifestyle and help it become incredible since it deserves. Our Specialist Gurgaon Escorts are experts in entertaining their buyers Other than pleasing their dick. They know around than 50 unique shades of coupling and ought to carry them into reading more Participate in although appeasing your starvation.

    ReplyDelete
  17. If you continue to provide the same nature of escort services, you must ensure that there are some of the best agencies available in each breadth and length. There are benefits that are important when it comes to resorting to the accompanying services offered by the agency.
    Ludhiana Escorts
    Escorts in Udaipur
    Escorts in Gurgaon
    Escorts in Jodhpur
    Escorts in Noida
    Escorts in Dehradun
    Call Girls in Chandigarh

    ReplyDelete
  18. We offer the most affordable assistance because we comprehend consumers' desires. Our Ludhiana Escorts are very friendly and talk with you quite perfectly so that you come to feel far closer to them. Change down the music when Ludhiana Escorts start out talking along with you. The extended travel also can make her feel enjoyable for your evening. Finish your experience in the cafe or resort and acquire a home way too! The Ludhiana younger ladies dealing with Cherry Ludhiana Escorts are seemingly Amongst the number of most effective non-public women that The entire city provides to your desk.

    ReplyDelete
  19. Our Escorts and services are very sincere and transparent to ensure our customers get to be aware of Each individual and each component regarding our Call Girls in Goa solutions. We're simply approachable. With the help with the Make contact with details talked about on our Web-site, a consumer can reach us whenever. We often enhance Call Girls in Goa companies and always desire to include many more facilities to please our clients. Our Call Girls in Goa provide the girlfriend or soulmate type encounter to our clients. The beauty of our Goa Escorts can help you to forget all the things regarding your worry.

    ReplyDelete
  20. Looking for elite Malviya Nagar Escorts Call Girls Service? Hire only genuine Malviya Nagar escorts girls through escorts Services available Incall & Outcalls 24/7.
    Malviya Nagar Escorts

    ReplyDelete
  21. With the gathering of charming and hot Call Girls in Goa, reputed wonderful escort agencies stand for their legitimate profiles of models. As it typically develops into difficult to refine the correct variety from all genuine Escort Service in Goa simply because all appears extremely loving within their proficient Photograph shoots. Housewives Call Girls in Goa are secured and assure you actual pleasure with complete fulfillment. I can bet if you are seeking a different experiment and need to learn more sex instruction select Housewives Call Girls in Goa.

    ReplyDelete
  22. The article shares very meaningful information, at least for me. Thanks for sharing, my friend! Please visit my website hope you enjoy it
    Escort Service in Nashik | Call girl in Nashik | Escorts in Nashik | Nashik Escorts

    ReplyDelete
  23. I am very Impressed to read this. This is the kind of manual that needs to be given and not the random misinformation that’s at the other blogs. Appreciate your sharing. Keep posting again
    Dwarka Escorts

    ReplyDelete
  24. You can choose to visit our Udaipur Call girls or reserve them online and call them at your location. As an authorized agency, we guarantee the security of all the details that our clients share with us. Therefore, you should not be concerned while revealing your identity or sharing your contact details with us.
    Udaipur escorts
    Jaipur escort service
    Goa escorts
    Hyderabad escorts
    Ahemadabad escorts
    Jodhpur escort service

    ReplyDelete
  25. If you are a tourist in Dehradun, don't miss the only opportunity to spend time with our Dehradun Call girls. By choosing our girls as partners, you can eliminate your loneliness that you can feel like a tourist in an unknown city. Our girls will make sure to give you satisfaction to leave you with an experience that will always remain with you in your memories.
    Dehradun Call Girls
    Rishikesh escort service
    Mussoorie escort service
    Haridwar escort service
    Nainital escort service
    Haldwani escort service

    ReplyDelete
  26. We make sure that all of our Nainital escort service undergo regular beauty treatments. Our beautiful ladies understand the needs of each client and act according to their preferences. As a company that always wants to provide the best services, we offer premium services to customers who wish to have them.
    Nainital escort service
    Haldwani escort service
    Escorts in Nainital
    Rudrapur escort service
    Ramnagar escort service

    ReplyDelete
  27. You can also choose to book our Jodhpur escort girls online and invite them to your preferred location. We always have special offers for clients who opt for our premium escort services. As escorts, our ladies will make sure you have a splendid evening with them.
    Jodhpur escort service
    Ajmer escort service
    Kishangarh escort service
    Pushkar Escort service
    Nasirabad escort service
    Beawar escort service

    ReplyDelete
  28. Hello Guys, I am Muskan Khanna, well educated & working as night model girl in Chennai Escorts, I am here to help and provide safe, secure & Independent Chennai Escorts Services. Call me at +91 99156 22743 for an appointment and to choose girls.

    ReplyDelete
  29. Let us call you call girls in Manesar, you will recognize that our women are every careful and secretive. Our Manesar escorts comes in personal first class to verify your requirements.Escort In Manesar

    ReplyDelete
  30. We provide you online service of Rohtak call girls. High profile escorts is the official portal.Escort Service In Rohtak

    ReplyDelete
  31. Hi Fi Model Escorts DLF offers high profile attractive independent escorts services in Gurgaon. You can contact us on our website for lovely cute girls.escort service in gurgaon

    ReplyDelete


  32. The agency follows a strict selection procedure and each and every female interested in joining us need to meet the essential conditions and fulfill the eligibility criteria. Call Girls in Gurgaon we have and There is a long list of elite customers who has blind faith in us. It is because of the uncompromising terms and conditions that we strictly follow to ensure highest level of privacy. Check our other Services...
    Escorts in Gurgaon
    Escorts Service in Gurgaon
    Female Escorts in Gurgaon
    Russian Call Girls in Gurgaon
    Russian Escorts in Gurgaon



    ReplyDelete
  33. If you are in Bhiwadi and you want to serve the call girls, then we provide you the service of cheap call girls and we provide you 24x7 hours of service.Bhiwadhi Escort

    ReplyDelete
  34. Welcome to Gurgaon Escorts Agency, we give you complete discount of receiving Russian and erotic call girls in Gurgaon, you can contact us on our website to get them. Call Girls In Gurgaon With Whatsapp Number

    ReplyDelete
  35. When it comes to dating professional Call Girls In Delhi With Photo and Mobile Number, there are many aspects which you should consider as escort dating is not as simple as it seems. Needless to mention, it is more about having erotic fun with the Cheap Girls.Female Escort in Delhi With Photo

    ReplyDelete
  36. Want to experiment pleasure and excitement in life? We have the perfect answer to your imagination! Enjoy the warm company of Call Girls in Delhi With Photo and take your senses to a new level of pleasure. When it comes to providing best Delhi girls, Mobile no one does it better than us. Delhi Escorts With Photos

    ReplyDelete