Monday, April 29, 2013

How to re-IP a Cloudera-Managed Cluster

[CM/CDH 4 // CentOS]

At work I was recently asked to install a Cloudera-managed cluster running Hadoop/Zookeeper/HBase at a location with no internet reachout. There were two options: I could perform a complete offline install at the customer site, or I could (save a boatload of time and) install online at our site and then re-ip the cluster at the customer site. Naturally I chose the latter. Offline installs are complicated (http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM4Ent/4.5.1/Cloudera-Manager-Enterprise-Edition-Installation-Guide/cmeeig_topic_6_6.html?scroll=cmeeig_topic_6_6), which defeats half the purpose of using Cloudera in the first place, though you'd still get the nice web UI to manage all your services once you'd finished the manual installation.

Here's what you have to do to update the IP addresses of a Cloudera-managed cluster (assuming the hostnames didn't change):


  • Ensure that the Cloudera services on all nodes in the cluster are stopped.

    service cloudera-scm-agent stop  //on master and all workers

  • Change the static IPs of the nodes to the desired new IPs.


  • Edit /etc/cloudera-scm-agent/config.ini on all cloudera worker nodes to point to the updated cloudera manager IP.


  • Restart all cloudera services in the cluster.


Don't forget to update your DNS forward / reverse lookups and any other references to those static IPs you may have lurking around in config files.

And, voila! Now you can move on to more important things.

Wednesday, March 27, 2013

configuring passwordless ssh on linux centos 6

define two hosts:
h1- host that will be doing the passwordless SSHing
h2- host that will be SSHd into 

1. h1:  (as root) cd /root/.ssh
2. h1:  ssh-keygen, enter, enter, enter (use default filename and empty passphrase)
3. h1:  ssh-copy-id -i ~/.ssh/id_rsa.pub username@h2
4. test SSHing from h1 to h2 -- shouldn't ask for password