I finally bit the bullet and brought some new hard ware with the aim of setting up a small home lab for the purposes of learning hadoop. I set up a small 3 machine cluster whist at RBI (part of reed Elsevier) to have a look at ML and AI and As I have the time was interested in learning hadoop especially as in the past I worked on a billing system that used the map reduce programing paradime on PR1ME 750’s. (dont knock it back then 17 750’s running together was not to be sniffed at) .
There are lots of sites dedicated to building a home labs for Vmware/vshphere and Cisco but none for hadoop so I thought I ought to document the process here,
After debating finishing of my antec 900 build and running on that using vmware player I decided to use a popular microserver from HP as they where cheap around £120 for the base machine there is a £50 rebate until the end of dec.
It out of the box can take 4 stata drives and with tweaks can take two more in the top 5.25 bay a 3.5 and 2.5 inch in a suitable adapter.
The main aim is to run a small lab cluster of virtual hadoop machines so that I can both get to grips with hadoop and MR but also so I can get my hands dirty (instead of just uing a single vm) and understand how it all works under the hood and play with some of the newer hadoop features such as NN HA .
Yes I know that hadoops performance under virtual machines isn’t going to be very good this is primarily a learning environment not one you would use for doing useful work.
I have the machine booted and after a struggle managed to create a bootable ESXi usb stick. I need to by a bigger USB stick as I want to install the hypervisor on that leaving all the disks free for VM’s.
The next step is to buy more memory to take the system up to its max of 16GB and some more DASD (disk drives for you civilians)
Yes thats what i thought booted of the usb key tried install over the top as you are supposed to and it choked due to lack of memory
ESXi only needs a 1GB USB drive to boot from.
My microserver used to run ESXi from USB (installed from a USB CD drive). I now run FreeNAS natively on it, using it to provide storage to a pair of Intel NUCs.
Its system memory am trying a hack to get round the limit – though i need to by a pair of 8GB modules to take it up to16gig any how