国产一级a片免费看高清,亚洲熟女中文字幕在线视频,黄三级高清在线播放,免费黄色视频在线看

打開APP
userphoto
未登錄

開通VIP,暢享免費(fèi)電子書等14項(xiàng)超值服

開通VIP
Intellij IDEA中配置依賴 ---- Hadoop 普通 java 項(xiàng)目
  1. 創(chuàng)建普通Java項(xiàng)目

  2. 添加引用

    1. FileProject Structure...ModulesDependencies+Library...Java

    2. 選擇/usr/local/Cellar/hadoop/2.5.2/libexec/share/hadoop目錄下除了httpfs外的全部文件夾。   (Ubuntu is : /usr/local/hadoop/share/hadoop/  )

    3. Name可以隨便寫,例如”common”,OK。

    4. +Jars or directories...

    5. 選擇/usr/local/Cellar/hadoop/2.5.2/libexec/share/hadoop/common/lib    (Ubuntu is : /usr/local/hadoop/share/hadoop/common/lib )

    此時(shí)Dependencies內(nèi)應(yīng)該總共增加了一個(gè)”common”和一個(gè)”lib”目錄。

  3. 修改Project Structure中的Artifacts,增加Jar包的生成配置。

生成HelloHadoop jar包

生成jar包的過(guò)程也比較簡(jiǎn)單,
1.選擇菜單File->Project Structure,彈出Project Structure的設(shè)置對(duì)話框。
2.選擇左邊的Artifacts后點(diǎn)擊上方的“+”按鈕
3.在彈出的框中選擇jar->from moduls with dependencies..
4.選擇要啟動(dòng)的類,然后 確定
5.應(yīng)用之后,對(duì)話框消失。在IDEA選擇菜單Build->Build Artifacts,選擇Build或者Rebuild后即可生成,生成的jar文件位于工程項(xiàng)目目錄的out/artifacts下。

(這樣對(duì)于LInux應(yīng)該就可以了,但是對(duì)Mac OS x是不行的。因?yàn)镺SX by default, the filesystem is set to case-insensitive . 一個(gè)解決方法是 轉(zhuǎn)用maven 來(lái)build project. 然后



可以寫代碼編譯打包了。



編寫WordCount MapReduce程序

這里直接使用了官方的代碼

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
import java.io.IOException;
import java.util.*;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapreduce.*;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
public class WordCount {
public static class Map extends Mapper<LongWritable, Text, Text, IntWritable> {
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(LongWritable key, Text value, Context context)
throws IOException, InterruptedException {
String line = value.toString();
StringTokenizer tokenizer = new StringTokenizer(line);
while (tokenizer.hasMoreTokens()) {
word.set(tokenizer.nextToken());
context.write(word, one);
}
}
}
public static class Reduce extends Reducer<Text, IntWritable, Text, IntWritable> {
public void reduce(Text key, Iterable<IntWritable> values, Context context)
throws IOException, InterruptedException {
int sum = 0;
for (IntWritable val : values) {
sum += val.get();
}
context.write(key, new IntWritable(sum));
}
}
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
Job job = new Job(conf, "wordcount");
job.setJarByClass(WordCount.class); //注意,必須添加這行,否則hadoop無(wú)法找到對(duì)應(yīng)的class
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
job.setMapperClass(Map.class);
job.setReducerClass(Reduce.class);
job.setInputFormatClass(TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.waitForCompletion(true);
}
}

需要注意的是需要在官方代碼中加入job.setJarByClass(WordCount.class);這一行,具體解釋可以參考這里。


運(yùn)行HelloHadoop jar包

將生成的HelloHadoop.jar傳送到hadoop集群的Name node節(jié)點(diǎn)上。
處于測(cè)試目的,簡(jiǎn)單寫了一個(gè)測(cè)試數(shù)據(jù)文本wctest.txt

1
2
3
4
this is hadoop test string
hadoop hadoop
test test
string string string

將該測(cè)試文本傳到HDFS

1
2
[hdfs@172-22-195-15 data]$ hdfs dfs -mkdir /user/chenbiaolong/wc_test_input
[hdfs@172-22-195-15 data]$ hdfs dfs -put wctest.txt /user/chenbiaolong/wc_test_input

cd 到j(luò)ar包對(duì)應(yīng)的目錄,執(zhí)行HelloHadoop jar包

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
[hdfs@172-22-195-15 code]$ cd WorkCount/
[hdfs@172-22-195-15 WorkCount]$ ls
HelloHadoop.jar
[hdfs@172-22-195-15 WorkCount]$ hadoop jar HelloHadoop.jar WordCount /user/chenbiaolong/wc_test_input /user/chenbiaolong/wc_test_output
15/03/26 15:54:19 INFO impl.TimelineClientImpl: Timeline service address: http://172-22-195-17.com:8188/ws/v1/timeline/
15/03/26 15:54:19 INFO client.RMProxy: Connecting to ResourceManager at 172-22-195-17.com/172.22.195.17:8050
15/03/26 15:54:20 WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
15/03/26 15:54:20 INFO input.FileInputFormat: Total input paths to process : 1
15/03/26 15:54:21 INFO mapreduce.JobSubmitter: number of splits:1
15/03/26 15:54:21 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1427255014010_0005
15/03/26 15:54:21 INFO impl.YarnClientImpl: Submitted application application_1427255014010_0005
15/03/26 15:54:21 INFO mapreduce.Job: The url to track the job: http://172-22-195-17.com:8088/proxy/application_1427255014010_0005/
15/03/26 15:54:21 INFO mapreduce.Job: Running job: job_1427255014010_0005
15/03/26 15:54:28 INFO mapreduce.Job: Job job_1427255014010_0005 running in uber mode : false
15/03/26 15:54:28 INFO mapreduce.Job: map 0% reduce 0%
15/03/26 15:54:34 INFO mapreduce.Job: map 100% reduce 0%
15/03/26 15:54:41 INFO mapreduce.Job: map 100% reduce 100%
15/03/26 15:54:42 INFO mapreduce.Job: Job job_1427255014010_0005 completed successfully
15/03/26 15:54:43 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=150
FILE: Number of bytes written=225815
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=210
HDFS: Number of bytes written=37
HDFS: Number of read operations=6
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=1
Launched reduce tasks=1
Rack-local map tasks=1
Total time spent by all maps in occupied slots (ms)=4133
Total time spent by all reduces in occupied slots (ms)=4793
Total time spent by all map tasks (ms)=4133
Total time spent by all reduce tasks (ms)=4793
Total vcore-seconds taken by all map tasks=4133
Total vcore-seconds taken by all reduce tasks=4793
Total megabyte-seconds taken by all map tasks=16928768
Total megabyte-seconds taken by all reduce tasks=19632128
Map-Reduce Framework
Map input records=4
Map output records=12
Map output bytes=120
Map output materialized bytes=150
Input split bytes=137
Combine input records=0
Combine output records=0
Reduce input groups=5
Reduce shuffle bytes=150
Reduce input records=12
Reduce output records=5
Spilled Records=24
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=91
CPU time spent (ms)=3040
Physical memory (bytes) snapshot=1466998784
Virtual memory (bytes) snapshot=8678326272
Total committed heap usage (bytes)=2200961024
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=73
File Output Format Counters
Bytes Written=37
[hdfs@172-22-195-15 WorkCount]$

結(jié)果被輸出到/user/chenbiaolong/wc_test_output

1
2
3
4
5
6
7
8
9
10
11
[hdfs@172-22-195-15 WorkCount]$ hdfs dfs -ls /user/chenbiaolong/wc_test_output
Found 2 items
-rw-r--r-- 3 hdfs hdfs 0 2015-03-26 15:54 /user/chenbiaolong/wc_test_output/_SUCCESS
-rw-r--r-- 3 hdfs hdfs 37 2015-03-26 15:54 /user/chenbiaolong/wc_test_output/part-r-00000
[hdfs@172-22-195-15 WorkCount]$ hdfs dfs -cat /user/chenbiaolong/wc_test_output/part-r-00000
hadoop 3
is 1
string 4
test 3
this 1
[hdfs@172-22-195-15 WorkCount]$

可以看出我們已經(jīng)順利得到正確結(jié)果。

本站僅提供存儲(chǔ)服務(wù),所有內(nèi)容均由用戶發(fā)布,如發(fā)現(xiàn)有害或侵權(quán)內(nèi)容,請(qǐng)點(diǎn)擊舉報(bào)。
打開APP,閱讀全文并永久保存 查看更多類似文章
猜你喜歡
類似文章
Sqoop1.4.4實(shí)現(xiàn)關(guān)系型數(shù)據(jù)庫(kù)多表同時(shí)導(dǎo)入HDFS或Hive中
本文將大數(shù)據(jù)學(xué)習(xí)門檻降到了地平線
配置hadoop
Hadoop API使用
精通HADOOP(五) - 初識(shí)Hadoop - 執(zhí)行和測(cè)試Hadoop樣例程序
Hadoop系列之(一):Hadoop單機(jī)部署
更多類似文章 >>
生活服務(wù)
分享 收藏 導(dǎo)長(zhǎng)圖 關(guān)注 下載文章
綁定賬號(hào)成功
后續(xù)可登錄賬號(hào)暢享VIP特權(quán)!
如果VIP功能使用有故障,
可點(diǎn)擊這里聯(lián)系客服!

聯(lián)系客服