Setting up a home hadoop lab #1 starting out

I finally bit the bullet and brought some new hard ware with the aim  of setting up a small home lab  for the purposes of learning hadoop. I set up a small 3 machine cluster whist at RBI (part of reed Elsevier) to have a look at ML and AI and As I have the time was interested in learning hadoop especially as in the past I worked on a billing system that used the map reduce programing paradime on PR1ME 750’s. (dont knock it back then 17 750’s running together was not to be sniffed at) .

A Prime 750

There are lots of sites dedicated to building a home labs for Vmware/vshphere  and Cisco but none for hadoop so I thought I ought to document the process here,

After debating finishing of my antec 900 build and running on that using vmware player  I decided to use a popular microserver from  HP as they where cheap around £120 for the base machine there is a £50 rebate until the end of dec.

It out of the box can take 4 stata drives and with tweaks can take two more in the top 5.25 bay a 3.5 and 2.5 inch in a suitable adapter.

The main aim is to run a small lab cluster of virtual hadoop machines so that I can both get to grips with hadoop and MR but also so I can get my hands dirty (instead of just uing a single vm) and  understand how it all works under the hood and play with some of the newer hadoop features such as NN HA .

Yes I know that hadoops performance under virtual machines isn’t going to be very good this is primarily a learning environment not one you would use for doing useful work.

I have the machine booted and after a struggle  managed to create a bootable ESXi usb stick. I need to by a bigger USB stick as I want to install the hypervisor on that leaving all the disks free for VM’s.

The next step is to buy more memory to take the system up to its max of 16GB and some more DASD (disk drives for you civilians)