當企鵝龍遇上小飛象 DRBL-Hadoop Jazz Wang Yao-Tsung Wang jazz@nchc.org.tw

Programmer v.s. System Admin. Source:http://www.funnyjunksite.com/wp-content/uploads/2007/08/programmer.jpg Source: http://www.sysadminday.com/images/people/136-3697.JPG

PART 1 : PART 2 : PART 3 : What is Cluster Computing ? Agenda PART 1 : What is Cluster Computing ? How to deploy PC cluster ? PART 2 : What is DRBL and Clonezilla ? Can DRBL help to deploy Hadoop ? PART 3 : Live Demo of DRBL Live and Clonezilla Live

PART 1 : PC Cluster 101 Jazz Wang Yao-Tsung Wang jazz@nchc.org.tw

At First, We have “ 4 + 1 ” PC Cluster It'd better be 2n Manage Scheduler

Then, We connect 5 PCs with Gigabit Ethernet Switch GiE Switch 10/100/1000 MBps WAN Add 1 NIC for WAN

Compute Nodes 4 Compute Nodes will communicate via LAN Switch. Only Manage Node have Internet Access for Security! LAN Switch WAN Manage Node

Basic System Setup for Cluster Compute Nodes Basic System Setup for Cluster Messaging Account Mgnt. MPICH SSHD NIS YP GCC GNU Libc Bash Kernel Module Perl Linux Kernel Boot Loader

We need to install Scheduler and Network File System for sharing On Manage Node, We need to install Scheduler and Network File System for sharing Files with Compute Node Job Mgnt. Messaging Account Mgnt. OpenPBS MPICH SSHD NIS YP File Sharing GCC GNU Libc NFS Bash Kernel Module Perl Extra Linux Kernel Boot Loader

Research topics about PC Cluster Process Architecture Storage Architecture System Architecture Network Architecture System-level Middleware Cluster Computing Parallel Computing Share Memory Programming Distributed Memory Programming Parallel Algorithms And Applications Application-level Middleware Programming Ref: Cluster Computing in the Classroom: Topics, Guidelines, and Experiences http://www.gridbus.org/papers/CC-Edu.pdf

Challenges of Cluster Computing Hardware Ethernet Speed / PC Density Power / Cooling / Heat Network and Storage Architecture Software Job Scheduler ( Cluster level ) Account Management File Sharing / Package Management Limitation Shared Memory Global Memory Management

Common Method to deploy Cluster 3. Configure Settings ↓ 4. Install Job Scheduler 5. Running Benchmark 2. Cloning to multiple machine 1. Setup one Template machine

Challenges of Common Method Add New User Account ? Upgrade Software ? How to share user data ? Configuration Syncronization

How to deploy 4000+ Nodes ????

Advanced Methods to deploy Cluster SSI ( Single System Image ) Multiple PCs as Single Computing Resources Image-based homogeneous ex. SystemImager, OSCAR, Kadeploy Package-based heterogeneous easy update and modify packages ex. FAI, DRBL Other deploy tools Rocks : RPM only cfengine : configuration engine

Comparison of Cluster Deploy Tools Distribution Support Diskless/ Sysmless Type Node configuration tools Cluster management Database installation System Imager ALL Yes Image No OSCAR RPM-based Kadeploy FAI Debian-Based Package

Hadoop Deployment Tool PART 2-1 : Hadoop Deployment Tool Jazz Wang Yao-Tsung Wang jazz@nchc.org.tw

Source: Deploying hadoop with smartfrog http://people.apache.org/~stevel/slides/deploying_hadoop_with_smartfrog.pdf

Source: Deploying hadoop with smartfrog http://people.apache.org/~stevel/slides/deploying_hadoop_with_smartfrog.pdf

Source: Deploying hadoop with smartfrog http://people.apache.org/~stevel/slides/deploying_hadoop_with_smartfrog.pdf

Source: Deploying hadoop with smartfrog http://people.apache.org/~stevel/slides/deploying_hadoop_with_smartfrog.pdf

Source: Deploying hadoop with smartfrog http://people.apache.org/~stevel/slides/deploying_hadoop_with_smartfrog.pdf

Source: Deploying hadoop with smartfrog http://people.apache.org/~stevel/slides/deploying_hadoop_with_smartfrog.pdf

Source: Deploying hadoop with smartfrog http://people.apache.org/~stevel/slides/deploying_hadoop_with_smartfrog.pdf

Source: Deploying hadoop with smartfrog http://people.apache.org/~stevel/slides/deploying_hadoop_with_smartfrog.pdf

Source: Deploying hadoop with smartfrog http://people.apache.org/~stevel/slides/deploying_hadoop_with_smartfrog.pdf

PART 2-2 : 企鵝龍與再生龍 工商服務時間 Jazz Wang Yao-Tsung Wang jazz@nchc.org.tw

何謂企鵝龍DRBL ?? = + + Server Diskless Remote Boot in Linux Diskfull PC 網路是便宜的，人的時間才是昂貴的。 企鵝龍簡單來說就是..... 用網路線取代硬碟排線 所有學生的電腦都透過網路連接到一台伺服器主機 Diskfull PC = + + Diskless PC Server source: http://www.mren.com.tw

何謂再生龍Clonezilla ?? Clone (複製) + zilla = Clonezilla (再生龍) 裸機備分還原工具 Norton Ghost 的自由軟體版替代方案 Disk to Disk Disk to Image Image to N Disks

降低資訊教育管理成本 需要「化繁為簡」的解決方案！ 人力、時間成本高 設備維護成本高 教師1人維護管理多組設備 教學同時分派或收集作業 需分別處理設定(每班約40台) 如：電腦中毒、環境設定 系統操作問題、開關機、 備份還原等 一般國內小學的電腦教室

平衡商業軟體與知識教育 知識和軟體都需要讓孩子「帶著走」！ 商業軟體授權高成本 知識與法治的學習 在校學習，也需回家複習 學校每台(平均) 2萬 學生家用(平均) 4萬 知識與法治的學習 教育知識，也需教育尊重 尊重智財權觀念

國網中心自由軟體開發 多元化資訊教學的新選擇！ 企鵝龍DRBL 再生龍Clonezilla 以個人叢集電腦(PC Cluster)經驗發展DRBL&Clonezilla 企鵝龍DRBL 再生龍Clonezilla (Diskless Remote Boot in Linux ) 適合將整個電腦教室轉換 成純自由軟體環境 適用完整系統備份、裸機 還原或災難復原 是自由！不是免費… 分送、修改、存取、使用軟體的自由。免費是附加價值。

企鵝龍DRBL與再生龍Clonezilla 電腦教室管理的新利器！ ■以每班40台電腦為估算單位

NT. 98,595,000 元 節省龐大軟體授權費 降低台灣盜版率 提升台灣形象 降低管理維護成本 帶動自由軟體使用 節樽軟體授權成本(估計) NT. 98,595,000 元 以某商業獨家軟體每機3000元授權費計，每班35台電腦(3000*35*939) 教育單位採用DRBL 高速計算研究 資料儲存備援 擴至全國各單位

PART 1-3 : 企鵝龍的開機原理 Jazz Wang Yao-Tsung Wang jazz@nchc.org.tw

Redhat, Fedora, CentOS, Mandriva, 1st, We install Base System of GNU/Linux on Management Node. You can choose: Redhat, Fedora, CentOS, Mandriva, Ubuntu, Debian, ... Linux Kernel Kernel Module GNU Libc Boot Loader

2nd, We install DRBL package and configure it as DRBL Server. There are lots of service needed: SSHD, DHCPD, TFTPD, NFS Server, NIS Server, YP Server ... DHCPD TFTPD NFS Bash Perl Network Booting YP NIS Account Mgnt. DRBL Server based on existing Open Source and keep Hacking! SSHD Linux Kernel Kernel Module GNU Libc Boot Loader

After running “drblsrv -i” & “drblpush -i”, there will be pxelinux, vmlinux-pex, initrd-pxe in TFTPROOT, and different configuration files for each Compute Node in NFSROOT NFS TFTPD DHCPD SSHD NIS YP pxelinux vmlinuz-pxe initrd-pxe Config. Files Ex. hostname Linux Kernel Kernel Module GNU Libc Boot Loader

3nd, We enable PXE function in BIOS configuration. BIOS PXE BIOS PXE BIOS PXE BIOS PXE NFS TFTPD DHCPD SSHD NIS YP pxelinux vmlinuz-pxe initrd-pxe Config. Files Ex. hostname Linux Kernel Kernel Module GNU Libc Boot Loader

While Booting, PXE will query IP address from DHCPD. BIOS PXE BIOS PXE BIOS PXE BIOS PXE NFS TFTPD DHCPD SSHD NIS YP pxelinux vmlinuz-pxe initrd-pxe Config. Files Ex. hostname Linux Kernel Kernel Module GNU Libc Boot Loader

While Booting, PXE will query IP address from DHCPD. IP 1 IP 2 IP 3 IP 4 NFS TFTPD DHCPD SSHD NIS YP pxelinux vmlinuz-pxe initrd-pxe Config. Files Ex. hostname Linux Kernel Kernel Module GNU Libc Boot Loader

After PXE get its IP address, it will download booting files from TFTPD. NFS TFTPD DHCPD SSHD NIS YP Config. Files Ex. hostname Linux Kernel Kernel Module GNU Libc Boot Loader initrd-pxe vmlinuz-pxe pxelinux

pxelinux vmlinuz initrd IP 1 IP 2 IP 3 IP 4 NFS TFTPD DHCPD SSHD NIS YP Config. Files Ex. hostname Linux Kernel Kernel Module GNU Libc Boot Loader initrd-pxe vmlinuz-pxe pxelinux

initrd initrd initrd initrd vmlinuz vmlinuz vmlinuz vmlinuz pxelinux pxelinux pxelinux pxelinux IP 1 IP 2 IP 3 IP 4 NFS TFTPD DHCPD SSHD NIS YP Config. Files Ex. hostname Linux Kernel Kernel Module GNU Libc Boot Loader After downloading booting files, scripts in initrd-pxe will config NFSROOT for each Compute Node. initrd-pxe vmlinuz-pxe pxelinux

Config. 1 Config. 2 Config. 3 Config. 4 initrd initrd initrd initrd vmlinuz vmlinuz vmlinuz vmlinuz pxelinux pxelinux pxelinux pxelinux IP 1 IP 2 IP 3 IP 4 NFS TFTPD DHCPD SSHD NIS YP Config. Files Ex. hostname Linux Kernel Kernel Module GNU Libc Boot Loader initrd-pxe vmlinuz-pxe pxelinux

Applications and Services will also deployed to each Compute Node Bash Perl SSHD Applications and Services will also deployed to each Compute Node via NFS .... NFS TFTPD DHCPD SSHD NIS YP Perl Bash DRBL Server

With the help of NIS and YP, You can login each Compute Node SSHD SSH Client With the help of NIS and YP, You can login each Compute Node with the Same ID / PASSWORD stored in DRBL Server! NFS TFTPD DHCPD SSHD NIS YP DRBL Server

PART 2 -1: 當企鵝龍遇上小飛象 Jazz Wang Yao-Tsung Wang jazz@nchc.org.tw

使用DRBL佈署Hadoop 仍在開發中，待整理套件 drbl-hadoop – 掛載本機硬碟給 HDFS 用 svn co http://trac.nchc.org.tw/pub/grid/drbl-hadoop hadoop-register – 註冊網站與ssh applet svn co http://trac.nchc.org.tw/pub/cloud/hadoop-register

關於hadoop.nchc.org.tw DRBL Server - 1台(hadoop)，加大/home與/tftpboot空間。 DRBL Client - 19台(hadoop101~hadoop119) 使用Cloudera的Debian套件 使用drbl-hadoop 的設定跟init.d script來協助部署 使用hadoop-register 來提供使用者註冊與ssh applet介面

Lesson Learn Cloudera套件的好處：使用init.d script 來啟動關閉 name node, data node, job tracker, task tracker 建立大量帳號： 可透過DRBL內建指令完成 /opt/drbl/sbin/drbl-useradd 使用者預設HDFS家目錄 跑迴圈切換使用者，下 hadoop fs -mkdir tmp 設定使用者HDFS權限 跑迴圈切換使用者，下 hadoop dfs -chown $(id) /usr/$(id) HDFS會使用/var/lib/hadoop/cache/hadoop/dfs MapReduce會使用/var/lib/hadoop/cache/hadoop/mapred

PART 2 -2: Live Demo Jazz Wang Yao-Tsung Wang jazz@nchc.org.tw

WAN DRBL-Live

Demo with DRBL-Live CD 1. Boot Server with DRBL-Live CD http://free.nchc.org.tw/drbl-live/stable/ 2. Download DRBL-Hadoop Script http://classcloud.org/drbl-hadoop-live.sh http://classcloud.org/drbl-hadoop-live-run.sh 3. Follow the steps http://classcloud.org/drbl-hadoop

Questions? Jazz Wang Yao-Tsung Wang jazz@nchc.org.tw