Unit 06 雲端分散式Hadoop實驗 -II

Presentation on theme: "Unit 06 雲端分散式Hadoop實驗 -II"— Presentation transcript:

1 Unit 06 雲端分散式Hadoop實驗 -II
M. S. Jian Department of Computer Science and Information Engineering National Formosa University Yunlin, Taiwan, ROC

2 Cloud Operating System - Unit 06: 雲端分散式Hadoop實驗 -II
首先要進行Java相關安裝 由於Java相關套件並不一定直接做為自由軟體包裝於Ubuntu的軟體庫中，因此需要針對套件庫作內容連結的更新 11/22/2018 Cloud Operating System - Unit 06: 雲端分散式Hadoop實驗 -II

3 Cloud Operating System - Unit 06: 雲端分散式Hadoop實驗 -II
首先針對套件來源路徑更新 sudo gedit(或vi) /etc/apt/sources.list 將出現所有套件路徑中的 tw.achieve.com 置換成ubuntu.stu.edu.tw 11/22/2018 Cloud Operating System - Unit 06: 雲端分散式Hadoop實驗 -II

4 Cloud Operating System - Unit 06: 雲端分散式Hadoop實驗 -II
安裝Java sudo apt-get purge java-gcj-compat sudo add-apt-repository "deb lucid partner" sudo apt-get update 以上會因為Ubuntu不同而有 不同的安裝結果訊息 11/22/2018 Cloud Operating System - Unit 06: 雲端分散式Hadoop實驗 -II

5 Cloud Operating System - Unit 06: 雲端分散式Hadoop實驗 -II
安裝Java(2) sudo apt-get install sun-java6-jdk sun-java6-plugin sudo update-java-alternatives -s java-6-sun 以上為sun Java安裝 sudo apt-get install openjava-6-jdk 以上為OpenJava安裝----- 11/22/2018 Cloud Operating System - Unit 06: 雲端分散式Hadoop實驗 -II

6 Cloud Operating System - Unit 06: 雲端分散式Hadoop實驗 -II
確認Java版本 安裝結束後可以確認Java版本 java –version OpenJava 須為6以上版本 Sun Java須為1.6以上版本 11/22/2018 Cloud Operating System - Unit 06: 雲端分散式Hadoop實驗 -II

7 Cloud Operating System - Unit 06: 雲端分散式Hadoop實驗 -II
安裝通訊連線 接著安裝ssh & rsync sudo apt-get install ssh rsync Hadoop在不同機器上運作時 會使用的通訊連線方式 11/22/2018 Cloud Operating System - Unit 06: 雲端分散式Hadoop實驗 -II

8 Cloud Operating System - Unit 06: 雲端分散式Hadoop實驗 -II
若已經是超級使用者 可以不使用sudo指令 先進入opt路徑 cd /opt 下載Hadoop套件 sudo wget hadoop rc1.tar.gz 解壓縮 sudo tar zxvf hadoop rc1.tar.gz 移動資料夾 sudo mv hadoop / hadoop 11/22/2018 Cloud Operating System - Unit 06: 雲端分散式Hadoop實驗 -II

9 Cloud Operating System - Unit 06: 雲端分散式Hadoop實驗 -II
設定群組與使用者 若已經是超級使用者 可以不使用sudo指令 創建群組 sudo addgroup hadoop 設定一個使用者名稱是Hadoop，在Hadoop群組下 sudo adduser --ingroup hadoop hadoop 更改資料夾權限 sudo chown -R hadoop:hadoop hadoop 或 chown -R hadoop /opt/hadoop 11/22/2018 Cloud Operating System - Unit 06: 雲端分散式Hadoop實驗 -II

10 Cloud Operating System - Unit 06: 雲端分散式Hadoop實驗 -II
若已經是超級使用者 可以不使用sudo指令 建立資料夾 sudo mkdir /var/hadoop 更改權限 sudo chown -R hadoop:hadoop /var/hadoop 11/22/2018 Cloud Operating System - Unit 06: 雲端分散式Hadoop實驗 -II

11 以Hadoop身分登入 su – hadoop 於opt路徑中，修改設定
sudo gedit(或vi) hadoop/conf/hadoop-env.sh 新增設定 export JAVA_HOME=/usr/lib/jvm/java-6-sun export HADOOP_HOME=/opt/hadoop export HADOOP_CONF_DIR=/opt/hadoop/conf 若使用OpenJava 路徑更改為 /usr/lib/jvm/java-6-openjdk 11/22/2018 Cloud Operating System - Unit 06: 雲端分散式Hadoop實驗 -II

12 Cloud Operating System - Unit 06: 雲端分散式Hadoop實驗 -II
修改core-site.xml sudo gedit(或vi) hadoop/conf/core-site.xml <configuration> <property> <name>fs.default.name</name> <value>hdfs://localhost:9000</value> </property> <name>hadoop.tmp.dir</name> <value>/var/hadoop/hadoop-${user.name}</value> </configuration> 11/22/2018 Cloud Operating System - Unit 06: 雲端分散式Hadoop實驗 -II

13 Cloud Operating System - Unit 06: 雲端分散式Hadoop實驗 -II
修改hdfs-site.xml sudo gedit(或vi) hadoop/conf/hdfs-site.xml <configuration> <property> <name>dfs.replication</name> <value>1</value> </property> </configuration> 11/22/2018 Cloud Operating System - Unit 06: 雲端分散式Hadoop實驗 -II

14 Cloud Operating System - Unit 06: 雲端分散式Hadoop實驗 -II
修改mapred-site.xml sudo gedit(或vi) hadoop/conf/mapred-site.xml <configuration> <property> <name>mapred.job.tracker</name> <value>localhost:9001</value> </property> </configuration> 11/22/2018 Cloud Operating System - Unit 06: 雲端分散式Hadoop實驗 -II

15 Cloud Operating System - Unit 06: 雲端分散式Hadoop實驗 -II
在Hadoop身分下，到/opt/hadoop路徑啟動 格式化 namenode bin/hadoop namenode -format 啟動Hadoop服務 bin/start-all.sh 查看啟動結果 jps 11/22/2018 Cloud Operating System - Unit 06: 雲端分散式Hadoop實驗 -II

16 Hadoop與MapReduce範例操作
wget 測試 bin/hadoop jar nchc-example.jar bin/hadoop jar nchc-example.jar hello 11/22/2018 Cloud Operating System - Unit 06: 雲端分散式Hadoop實驗 -II

17 實際練習Hadoop與MapReduce
Word Count 初級 mkdir lab4_input echo “I like NCHC Cloud Course.” > lab4_input/input1 echo “I like nchc Cloud Course, and we enjoy this course.” > lab4_input/input2 bin/hadoop fs -put lab4_input lab4_input bin/hadoop fs -ls lab4_input 11/22/2018 Cloud Operating System - Unit 06: 雲端分散式Hadoop實驗 -II

18 Cloud Operating System - Unit 06: 雲端分散式Hadoop實驗 -II
執行Word Count範例 於 wget Mkdir MyJava javac –classpath hadoop-core jar -d MyJava WordCount.java jar -cvf wordcount.jar -C MyJava . bin/hadoop jar wordcount.jar WordCount lab4_input/ lab4_out1/ bin/hadoop fs -cat lab4_out1/part-r-00000 11/22/2018 Cloud Operating System - Unit 06: 雲端分散式Hadoop實驗 -II

19 Cloud Operating System - Unit 06: 雲端分散式Hadoop實驗 -II
Web網頁型態界面 - Hadoop 管理介面 - Hadoop Task Tracker 狀態 - Hadoop DFS 狀態 11/22/2018 Cloud Operating System - Unit 06: 雲端分散式Hadoop實驗 -II

20 Cloud Operating System - Unit 06: 雲端分散式Hadoop實驗 -II
範例 11/22/2018 Cloud Operating System - Unit 06: 雲端分散式Hadoop實驗 -II