jump to navigation

Setting up a home hadoop lab #1 starting out December 13, 2013

Posted by Neuromancer in hadoop.
Tags:
trackback

I finally bit the bullet and brought some new hard ware with the aim  of setting up a small home lab  for the purposes of learning hadoop. I set up a small 3 machine cluster whist at RBI (part of reed Elsevier) to have a look at ML and AI and As I have the time was interested in learning hadoop especially as in the past I worked on a billing system that used the map reduce programing paradime on PR1ME 750’s. (dont knock it back then 17 750’s running together was not to be sniffed at) .

r82r3196m

A Prime 750

There are lots of sites dedicated to building a home labs for Vmware/vshphere  and Cisco but none for hadoop so I thought I ought to document the process here,

After debating finishing of my antec 900 build and running on that using vmware player  I decided to use a popular microserver from  HP as they where cheap around £120 for the base machine there is a £50 rebate until the end of dec.

hp54gl-server

It out of the box can take 4 stata drives and with tweaks can take two more in the top 5.25 bay a 3.5 and 2.5 inch in a suitable adapter.

The main aim is to run a small lab cluster of virtual hadoop machines so that I can both get to grips with hadoop and MR but also so I can get my hands dirty (instead of just uing a single vm) and  understand how it all works under the hood and play with some of the newer hadoop features such as NN HA .

Yes I know that hadoops performance under virtual machines isn’t going to be very good this is primarily a learning environment not one you would use for doing useful work.

I have the machine booted and after a struggle  managed to create a bootable ESXi usb stick. I need to by a bigger USB stick as I want to install the hypervisor on that leaving all the disks free for VM’s.

The next step is to buy more memory to take the system up to its max of 16GB and some more DASD (disk drives for you civilians)

Comments»

1. Neuromancer - December 13, 2013

Yes thats what i thought booted of the usb key tried install over the top as you are supposed to and it choked due to lack of memory

2. lazyllama - December 13, 2013

ESXi only needs a 1GB USB drive to boot from.
My microserver used to run ESXi from USB (installed from a USB CD drive). I now run FreeNAS natively on it, using it to provide storage to a pair of Intel NUCs.

Neuromancer - December 14, 2013

Its system memory am trying a hack to get round the limit – though i need to by a pair of 8GB modules to take it up to16gig any how


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 391 other followers

%d bloggers like this: