Get Off Of My Cloud!

Posted: May 27, 2011 | Author: Sideb0ard | Filed under: abstraction, cloud, configuration management, future, hadoop, hardware, puppet, simulator, unix, virtualization, xen | Leave a comment »

I work around Hadoop a lot at work, however my duties don’t really require me to interact much beyond the occasional data-mining work in Hive. Recently, I was intrigued to read that Amazon’s EC2 services are built upon Xen Hypervisor, which is also something I use, but don’t administer. I figured I would combine them into learning exercise by setting up a Xen server, create a little virtual cluster, configure them all via Puppet (another project i’ve been wanting a look at) and finally setup Hadoop on them all. Here’s what happened..

XEN INSTALL——————–

So  - Xen is virtualisation software – the master host where the Hypervisor software is installed is called Dom0, and it can have many DomU guests, each a virtual machine which can run a host Operating System. In order to install the Xen Hypervisor and run Dom0, the kernel needs some modifications, and my first surprise was to discover recent releases of Ubuntu’s kernel don’t have Dom0 support. (The most recent discussion I could find on this was here, from 2010, so things may have changed)

My Ubuntu install has been through several years of upgrading and was getting a little crufty, plus the recent move to Unity desktop with Natty 11.04 wasn’t exciting me very much. I’ve also been keen to give Debian another go, as it’s been a few years, so rather than backport a Debian kernel to Ubuntu, I just went for a fresh install of Debian 6.0/Squeeze.

Aside from kernel support, more recent hardware support seems, if not essential, to make installation a lot easier. You need some BIOS support and CPU support – I’m running on a DELL Precision 390 – BIOS v.2.1.2 and CPU is a Pentium Core-2 Duo. I switched on Virtualization under the Performance section of my BIOS.

Aiight, so following instructions at http://wiki.debian.org/Xen i installed kernel and meta package..

apt-get install xen-linux-system

Then to ensure the xen kernel is loaded first, you have to move some files around:
cd /etc/grub.d/
mv 10_linux 50_linux
update-grub

Reboot machine and you should be running with a Dom0 kernel:

root@bitbot:/etc/puppet# uname -a
Linux bitbot 2.6.32-5-xen-amd64 #1 SMP Thu May 5 00:57:12 UTC 2011 x86_64 GNU/Linux

Now install Xen tools..

apt-get install xen-tools

Following some older HOWTO’s i started building my own filesystems for my first Guest machine:

dd if=/dev/zero of=/Xen/host/weebit1/root.img bs=1024k count=1024mkfs -t ext3 /Xen/hosts/weebit1/root.img
dd if=/dev/zero of=/Xen/host/weebit1/swap bs=1024k count=1024

However –  DON’T! there’s no need - Have a read through of the instructions here -

xen-create-image –manual

(basically it does everything for you – i.e. creating partitions & swap, formatting, filesystems, xen config file, installing OS)

Then all you need to type is:
xen-create-image –hostname weebit –vcpus 2 –scsi –dist squeeze–pygrub –dir=/Xen/host

(i had tried installing natty first, but was having trouble booting the VM - ran into issues with “ALERT!  /dev/sda2 does not exist.  Dropping to a shell!“)

With Debian systems, the Xen install doesn’t enable the Ethernet bridging by default so you need to edit /etc/xen/xend-config.sxp and uncomment:

(network-script network-bridge)

(Or when you try to start up your VM later you will run into:
“xm create /etc/xen/weebit.cfg -cError: Device 0 (vif) could not be connected. Could not find bridge, and none was specified“)

Restart Xend
/etc/init.d/xend restart

Once you have your Xen hosts created succesfully, you can start them with a command like:
xm create /etc/xen/weebit.cfg -c (where the cfg file matches the name of the host you created)

I created two VM’s – weebit and weebot – (and my master host is called bitbot) – yes, it gets confusing!

PUPPET SETUP..——————

Okay, two VMs setup with networking, so next thing was setting up Puppet on them – this part was by far the most troublesome – although there is nothing particularly complex about Puppet, it’s error messages aren’t the most verbose or helpful, and I found it quite sensitive to permissions and other peculiarities.

Here’s the main docs:
http://docs.puppetlabs.com/guides/configuring.html

This part is not essential (you can override it in configuration), however a default Puppet install expects the server to be called puppet so it’s best to setup some name resolution -I set it up simply via /etc/hosts.

PUPPET SERVER setup:

apt-get install puppet puppetmaster
update-rc.d puppetmaster defaults

Just to test it locally, run a puppet agent on the server:

puppet agent –test

All looks good, apart from it doesn’t do anything yet. My first test and following with many of the HOWTOs out there was to fix sudoers perms

vi manifests/site.pp

# fixup permissions on sudo
class sudo {
file { “/etc/sudoers”:
owner => root,
group => root,
mode => 440,
}
}

node default {
include sudo
}

In order to give it something to fix, i changed the ownership to my own user and altered the perm:
chown thorsten /etc/sudoers
chmod 400 /etc/sudoers

Ran this again:
puppet agent –test

Nothing! huh. however, running:
puppet apply /etc/puppet/manifests/site.pp

That worked fine. So first lesson – after any config changes – you need to restart Puppet -
/etc/init.d/puppermaster restart

puppet agent –test
notice: /Stage[main]/Sudo/File[/etc/sudoers]/owner: owner changed ‘thorsten’ to ‘root’

Boom! all good.

Now, rather than have a monolithic site manifest, Puppet best practises advocates a modular approach.

“Each module has a specific directory structure that allows Puppet to find all elements of the module and auto-load them.”

mkdir -p /etc/puppet/modules/sudo/manifests

move the ‘sudo’ class logic from site.pp to modules/sudo/manifests/init.pp:

vi modules/sudo/manifests/init.pp

# /etc/puppet/modules/sudo/manifests/init.pp
class sudo {
package { sudo: ensure => latest }
file { “/etc/sudoers”:
owner   => root,
group   => root,
mode    => 440,
source  => “puppet:///sudo/sudoers”,
require => Package["sudo"],
}
}

mkdir /etc/puppet/modules/sudo/files
cp /etc/sudoers /etc/puppet/modules/sudo/files

Modules in the modulepath should be auto loaded, however we can explicitly import them too:

# /etc/puppet/manifests/modules.pp
import “sudo”

Move the Node list into manifests/nodes.pp

node default {
include sudo
}

Finally, update site.pp to remove previous statements, and have it import the new modules and nodes .pp files:

# /etc/puppet/manifests/site.pp
import “modules”
import “nodes”

Again, restart Puppetmaster, then test the agent:

puppet agent –test
err: /Stage[main]/Sudo/File[/etc/sudoers]: Could not evaluate: Error 400 on SERVER: Permission denied – /etc/puppet/modules/sudo/files/sudoers Could not retrieve file metadata for puppet:///sudo/sudoers: Error 400 on SERVER: Permission denied – /etc/puppet/modules/sudo/files/sudoers at /etc/puppet/modules/sudo/manifests/init.pp:13

Second lesson! Make sure the modules dir is owned by your Puppet user account:

chown -R puppet modules

puppet agent –test

Cool, all good again!

PUPPET CLIENT INSTALL———-

apt-get install puppet
In order to have it start at boot:

vi /etc/default/puppet

Test it..
puppet agent –server puppet –waitforcert 60 –test

You should see:
info: Requesting certificate
warning: peer certificate won’t be verified in this SSL session
notice: Did not receive certificate

This is the inbuilt security, whereby the server needs to approve and sign the client certificates before they can be administered.

Back on the server type:
puppet cert –list

You should see your client machine listed. To approve it set:
puppet cert –sign weebot

BANG!

Now on your client, you can run:
puppet agent –test
(the –test option adds verbose and no-daemonize to the runtime so you can see whats going on)

You should hopefully see it now install the sudo package on your clients.

I wanted to test adding another package beyond the SUDO setup, so went ahead with NTP

cd /etc/puppet/modules/

mkdir -p ntp/manifests <– WATCH OUT, I ORIGINALLY NAMED THIS ‘MANIFEST’ – SINGULAR – AND IT TOOK ME AGES TO FIND OUT WHY IT WASN’T WORKING

vi ntp/manifests/init.pp
# /etc/puppet/modules/ntp/manifests/init.pp
class ntp {
package { “ntp”:
ensure => installed
}

service { “ntp”:
ensure => running,
}
}

Added ntp to manifests/modules with:
import “ntp”

updated manifests/nodes.pp with:
include ntp

And again -
chown -R puppet modules
/etc/init.d/puppermaster restart

On the clients:

puppet agent –test

So just to summarize – watch your permissions, watch yer typos and remember to restart!

For troubleshooting, puppetmaster logs to /var/log.syslog by default

I searched around for a while to find out whether I would need to set up cron jobs to automate the client, but no, if you set /etc/default/puppet to start at boot, as mentioned above, it will launch and daemonize itself. Default is to run every 30mins. To find out all the default options, run puppet agent –genconfig

HADOOP SETUP————————-

Okay, we have one master node, and two VM’s running, all configured with Puppet. Now to setup Hadoop on them.

There are no Hadoop packages in the standard Debian repositories, so you have to add Cloudera’s manually. They don’t have specific packages for Squeeze yet, but the Lenny ones work fine.

Add the following to your /etc/apt/sources.list on the Master:
#HADOOP
deb http://archive.cloudera.com/debian lenny-cdh3b3 contrib
deb-src http://archive.cloudera.com/debian lenny-cdh3b3 contrib

Then add their public key to your apt keyring:
curl -s http://archive.cloudera.com/debian/archive.key | sudo apt-key add -

apt-get update
and
apt-get install hadoop-0.20
to ensure that all works fine on the server.

All good?
aiiiight..

Okay, so we’ve been using Puppet to configure the VMs, so let’s also add apt.sources to our Puppet configuration:

cd /etc/puppet
mkdir -p modules/apt/manifests
mkdir -p modules/apt/files
cp /etc/apt/sources.list modules/apt/files/

Create:
# /etc/puppet/modules/apt/manifests/init.pp

class apt {

file { “/etc/apt/sources.list”:
owner => root,
group => root,
mode => 644,
source => “puppet:///apt/sources.list”,
}

exec { subscribe-echo:
command => “/usr/bin/apt-get -q -q update”,
logoutput => false,
refreshonly => true,
subscribe => file["/etc/apt/sources.list"]
}
}

chown -R puppet modules/

Add the new entries to:
vi manifests/modules.pp
import “apt”
vi manifests/nodes.pp
include apt
/etc/init.d/puppetmaster restart

At this point, I manually added the cloudera public key to both VMs (same command as above) – probably a smarter way of doing this, but this sufficed for my own efforts:
“curl -s http://archive.cloudera.com/debian/archive.key | sudo apt-key add -”

Again, run the:
puppet client –test
and if the planets are in alignment, your updated sources.list file should be copied over to the clients, and an apt-get update will be run.

Ok, so hopefully just one last step for Hadoop installation. Lets now add a Hadoop class to Puppet:

mkdir -p /etc/puppet/modules/hadoop/manifests
vi /etc/puppet/modules/hadoop/manifests/init.pp

# /etc/puppet/modules/hadoop/manifests/init.pp

class hadoop {

package { “hadoop-0.20″:
ensure => installed
}
}

The now familiar dance..
vi manifests/modules.pp
add import “hadoop”

vi manifests/nodes.pp
add include hadoop

chown -R puppet modules/
/etc/init.d/puppetmaster restart

Moment of truth.. Go to the clients and run our:
puppet agent –test

The following packages have unmet dependencies:
hadoop-0.20 : Depends: sun-java6-jre but it is not going to be installed
Depends: sun-java6-bin but it is not going to be installed

Bugger! You have to accept a user license to use Sun’s (well Oracle now) version of Java, so I had to run ‘apt-get -f install‘ by hand so i could accept Yes on the license. I’m not sure if there is an automated way of doing this or if it is necessary. Again, for my purposes it was fine to just do by hand. If you have a much bigger installation you might want to work out a workaround for this. ( I had already installed Java on my master/physical machine)

So – an almost automated install of Hadoop on a three node cluster! I haven’t configured my cluster with anything useful yet, so I’ll leave that for another article..

thor

#############

The following resources all proved useful :
http://docs.puppetlabs.com/references/latest/configuration.html
http://docs.puppetlabs.com/guides/tools.html
http://docs.puppetlabs.com/man/agent.html
http://bitfieldconsulting.com/puppet-tutorial
http://www.debian-administration.org/articles/526
http://projects.puppetlabs.com/projects/puppet/wiki/Simplest_Puppet_Install_Pattern
http://bitcube.co.uk/content/puppet-errors-explained
http://groups.google.com/group/puppet-users/browse_thread/thread/66361418d801a97c

---