性感美女污视频,快手美女视频集合,性感美女慢摇视频

創(chuàng)建普通Java項(xiàng)目
添加引用
1. File – Project Structure... – Modules – Dependencies –+ – Library... – Java
2. 選擇/usr/local/Cellar/hadoop/2.5.2/libexec/share/hadoop目錄下除了httpfs外的全部文件夾。 (Ubuntu is : /usr/local/hadoop/share/hadoop/ )
3. Name可以隨便寫，例如”common”，OK。
4. + – Jars or directories...
5. 選擇/usr/local/Cellar/hadoop/2.5.2/libexec/share/hadoop/common/lib (Ubuntu is : /usr/local/hadoop/share/hadoop/common/lib )
此時(shí)Dependencies內(nèi)應(yīng)該總共增加了一個(gè)”common”和一個(gè)”lib”目錄。
修改Project Structure中的Artifacts，增加Jar包的生成配置。

生成HelloHadoop jar包

生成jar包的過(guò)程也比較簡(jiǎn)單，
1.選擇菜單File->Project Structure，彈出Project Structure的設(shè)置對(duì)話框。
2.選擇左邊的Artifacts后點(diǎn)擊上方的“+”按鈕
3.在彈出的框中選擇jar->from moduls with dependencies..
4.選擇要啟動(dòng)的類，然后確定
5.應(yīng)用之后，對(duì)話框消失。在IDEA選擇菜單Build->Build Artifacts,選擇Build或者Rebuild后即可生成，生成的jar文件位于工程項(xiàng)目目錄的out/artifacts下。

(這樣對(duì)于LInux應(yīng)該就可以了，但是對(duì)Mac OS x是不行的。因?yàn)镺SX by default, the filesystem is set to case-insensitive . 一個(gè)解決方法是轉(zhuǎn)用maven 來(lái)build project. 然后

可以寫代碼編譯打包了。

編寫WordCount MapReduce程序

這里直接使用了官方的代碼

import java.io.IOException;
import java.util.*;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapreduce.*;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
public class WordCount {
    public static class Map extends Mapper<LongWritable, Text, Text, IntWritable> {
        private final static IntWritable one = new IntWritable(1);
        private Text word = new Text();
        public void map(LongWritable key, Text value, Context context) 
            throws IOException, InterruptedException {
            
            String line = value.toString();
            StringTokenizer tokenizer = new StringTokenizer(line);
            while (tokenizer.hasMoreTokens()) {
                word.set(tokenizer.nextToken());
                context.write(word, one);
            }
        }
    }
    public static class Reduce extends Reducer<Text, IntWritable, Text, IntWritable> {
        public void reduce(Text key, Iterable<IntWritable> values, Context context)
                throws IOException, InterruptedException {
            int sum = 0;
            for (IntWritable val : values) {
                sum += val.get();
            }
            context.write(key, new IntWritable(sum));
        }
    }
    public static void main(String[] args) throws Exception {
        Configuration conf = new Configuration();
        Job job = new Job(conf, "wordcount");
        job.setJarByClass(WordCount.class); //注意，必須添加這行，否則hadoop無(wú)法找到對(duì)應(yīng)的class
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);
        job.setMapperClass(Map.class);
        job.setReducerClass(Reduce.class);
        job.setInputFormatClass(TextInputFormat.class);
        job.setOutputFormatClass(TextOutputFormat.class);
        FileInputFormat.addInputPath(job, new Path(args[0]));
        FileOutputFormat.setOutputPath(job, new Path(args[1]));
        job.waitForCompletion(true);
    }
}

需要注意的是需要在官方代碼中加入job.setJarByClass(WordCount.class);這一行，具體解釋可以參考這里。

運(yùn)行HelloHadoop jar包

將生成的HelloHadoop.jar傳送到hadoop集群的Name node節(jié)點(diǎn)上。
處于測(cè)試目的，簡(jiǎn)單寫了一個(gè)測(cè)試數(shù)據(jù)文本wctest.txt

this is hadoop test string
hadoop hadoop
test test
string string string

將該測(cè)試文本傳到HDFS

1 2	[hdfs@172-22-195-15 data]$ hdfs dfs -mkdir /user/chenbiaolong/wc_test_input [hdfs@172-22-195-15 data]$ hdfs dfs -put wctest.txt /user/chenbiaolong/wc_test_input

cd 到j(luò)ar包對(duì)應(yīng)的目錄，執(zhí)行HelloHadoop jar包

[hdfs@172-22-195-15 code]$ cd WorkCount/
[hdfs@172-22-195-15 WorkCount]$ ls
HelloHadoop.jar  
[hdfs@172-22-195-15 WorkCount]$ hadoop jar HelloHadoop.jar WordCount /user/chenbiaolong/wc_test_input /user/chenbiaolong/wc_test_output
15/03/26 15:54:19 INFO impl.TimelineClientImpl: Timeline service address: http://172-22-195-17.com:8188/ws/v1/timeline/
15/03/26 15:54:19 INFO client.RMProxy: Connecting to ResourceManager at 172-22-195-17.com/172.22.195.17:8050
15/03/26 15:54:20 WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
15/03/26 15:54:20 INFO input.FileInputFormat: Total input paths to process : 1
15/03/26 15:54:21 INFO mapreduce.JobSubmitter: number of splits:1
15/03/26 15:54:21 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1427255014010_0005
15/03/26 15:54:21 INFO impl.YarnClientImpl: Submitted application application_1427255014010_0005
15/03/26 15:54:21 INFO mapreduce.Job: The url to track the job: http://172-22-195-17.com:8088/proxy/application_1427255014010_0005/
15/03/26 15:54:21 INFO mapreduce.Job: Running job: job_1427255014010_0005
15/03/26 15:54:28 INFO mapreduce.Job: Job job_1427255014010_0005 running in uber mode : false
15/03/26 15:54:28 INFO mapreduce.Job:  map 0% reduce 0%
15/03/26 15:54:34 INFO mapreduce.Job:  map 100% reduce 0%
15/03/26 15:54:41 INFO mapreduce.Job:  map 100% reduce 100%
15/03/26 15:54:42 INFO mapreduce.Job: Job job_1427255014010_0005 completed successfully
15/03/26 15:54:43 INFO mapreduce.Job: Counters: 49
        File System Counters
                FILE: Number of bytes read=150
                FILE: Number of bytes written=225815
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=210
                HDFS: Number of bytes written=37
                HDFS: Number of read operations=6
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=2
        Job Counters 
                Launched map tasks=1
                Launched reduce tasks=1
                Rack-local map tasks=1
                Total time spent by all maps in occupied slots (ms)=4133
                Total time spent by all reduces in occupied slots (ms)=4793
                Total time spent by all map tasks (ms)=4133
                Total time spent by all reduce tasks (ms)=4793
                Total vcore-seconds taken by all map tasks=4133
                Total vcore-seconds taken by all reduce tasks=4793
                Total megabyte-seconds taken by all map tasks=16928768
                Total megabyte-seconds taken by all reduce tasks=19632128
        Map-Reduce Framework
                Map input records=4
                Map output records=12
                Map output bytes=120
                Map output materialized bytes=150
                Input split bytes=137
                Combine input records=0
                Combine output records=0
                Reduce input groups=5
                Reduce shuffle bytes=150
                Reduce input records=12
                Reduce output records=5
                Spilled Records=24
                Shuffled Maps =1
                Failed Shuffles=0
                Merged Map outputs=1
                GC time elapsed (ms)=91
                CPU time spent (ms)=3040
                Physical memory (bytes) snapshot=1466998784
                Virtual memory (bytes) snapshot=8678326272
                Total committed heap usage (bytes)=2200961024
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        File Input Format Counters 
                Bytes Read=73
        File Output Format Counters 
                Bytes Written=37
[hdfs@172-22-195-15 WorkCount]$

結(jié)果被輸出到/user/chenbiaolong/wc_test_output

[hdfs@172-22-195-15 WorkCount]$ hdfs dfs -ls /user/chenbiaolong/wc_test_output
Found 2 items
-rw-r--r--   3 hdfs hdfs          0 2015-03-26 15:54 /user/chenbiaolong/wc_test_output/_SUCCESS
-rw-r--r--   3 hdfs hdfs         37 2015-03-26 15:54 /user/chenbiaolong/wc_test_output/part-r-00000
[hdfs@172-22-195-15 WorkCount]$ hdfs dfs -cat /user/chenbiaolong/wc_test_output/part-r-00000
hadoop  3
is      1
string  4
test    3
this    1
[hdfs@172-22-195-15 WorkCount]$

可以看出我們已經(jīng)順利得到正確結(jié)果。

本站僅提供存儲(chǔ)服務(wù)，所有內(nèi)容均由用戶發(fā)布，如發(fā)現(xiàn)有害或侵權(quán)內(nèi)容，請(qǐng)點(diǎn)擊舉報(bào)。

国产一级a片免费看高清,亚洲熟女中文字幕在线视频,黄三级高清在线播放,免费黄色视频在线看

生成HelloHadoop jar包

編寫WordCount MapReduce程序

運(yùn)行HelloHadoop jar包