Recent questions tagged Hadoop

0 votes

819 views

1 answer

hadoop - Connect to kerberised hive using jdbc from remote windows system

I have setup a hive environment with Kerberos security enabled on a Linux server (Red Hat). And I need ... answer will be appreciated. Thanks See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

792 views

1 answer

hadoop - csv reading in pig, csv file contains quoted comma

So my data looks something like this asdf, asdf, "adsf,qwef", asdf When i read that data in pig using ... write a pigscript to do this. See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

559 views

1 answer

hadoop - Why submitting job to mapreduce takes so much time in General?

So usually for 20 node cluster submitting job to process 3GB(200 splits) of data takes about 30sec and actual ... splitting 2. jar file sharing See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

751 views

1 answer

hadoop - How to handle spill memory in pig

My code like like this: pymt = LOAD 'pymt' USING PigStorage('|') AS ($pymt_schema); pymt_grp = GROUP ... and accumulator interface in my UDF. See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

721 views

1 answer

hadoop - remove surrounding quotes from fields while loading data into hive

I want to load a table with input data into hive. I have data in the following format. "153662";"0002241447 ... field as INT while making table. See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

831 views

1 answer

hadoop - Access files that start with underscore in apache spark

I am trying to access gz files on s3 that start with _ in Apache Spark. Unfortunately spark deems these files invisible ... .6.0 with Hadoop 2.3 See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

784 views

1 answer

hadoop - In spark join, does table order matter like in pig?

Related to Spark - Joining 2 PairRDD elements When doing a regular join in pig, the last table in the join ... scala Or am I missing something? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

877 views

1 answer

hadoop - Caused by: java.lang.ClassNotFoundException: org.apache.zookeeper.KeeperException

When I run a JAVA program(can not new HTable) under the Eclipse IDE, i got following errors: Exception in thread ... 0.jar Where is this wrong? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

768 views

1 answer

hadoop - Oozie > Java action > why property oozie.launcher.mapred.child.java.opts does not work

I am working on Oozie with a Java action. The Java action should use Java option -Xmx15g. Accordingly I set the ... Oozie is not a handy tool. See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

647 views

1 answer

hadoop - Too many open files in EMR

I am getting the following excpetion in my reducers: EMFILE: Too many open files at org.apache.hadoop.io.nativeio. ... is run on each slave? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

835 views

1 answer

hadoop - Apache Oozie failed loading ShareLib

i got the following oozie.log : org.apache.oozie.service.ServiceException: E0104: Could not fully initialize service [ ... idea of my mistake ? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

978 views

1 answer

hadoop - How to check whether the file exist in HDFS location, using oozie?

How to check whether a file in HDFS location is exist or not, using Oozie? In my HDFS location I will get a ... if the file exists in HDFS? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

615 views

1 answer

hadoop - Run a Local file system directory as input of a Mapper in cluster

I gave an input to the mapper from a local filesystem.It is running successfully from eclipse,But not running from ... get the output in hdfs See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.3k views

1 answer

hadoop - How to convert a Date String from UTC to Specific TimeZone in HIVE?

My Hive table has a date column with UTC date strings. I want to get all rows for a specific EST date. I am ... 01-12T15:53:00.000Z 1970-01-16 See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

841 views

1 answer

hadoop - Is it possible to read pdf/audio/video files(unstructured data) using Apache Spark?

Is it possible to read pdf/audio/video files(unstructured data) using Apache Spark? For example, I have ... do to process unstructured data? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

811 views

1 answer

hadoop - How to pass Hive conf variable in hive udf?

I want to pass hive conf variable to hive UDF. below is a code snippet. hive -f ../hive/testHive.sql ... access Hive configuration in hive UDF? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

978 views

1 answer

hadoop - Spark uses s3a: java.lang.NoSuchMethodError

I'm doing something about the combination of spark_with_hadoop2.7 (2.4.3), hadoop (3.2.0) and Ceph luminous. ... s3a throw when I invoke spark? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

749 views

1 answer

hadoop - Auto renewal of Kerberos ticket not working from Java

In my server application I'm connecting to Kerberos secured Hadoop cluster from my java application. On the ... in the above answer? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

741 views

1 answer

hadoop - How hashing works in bucketing for hive?

I know the hashing principal for HashMap in Java, so wanted to know that how the hashing works for the Hive ... the data in various bucket. See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

768 views

1 answer

hadoop yarn - Dataproc set number of vcores per executor container

I'm building a spark application which will run on Dataproc. I plan to use ephemeral clusters, and spin a new one ... and all the CPUs are used. See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

667 views

1 answer

hadoop - how to limit the number of mappers

I explicitly specify the number of mappers within my java program using conf.setNumMapTasks(), but when the job ... to the specified value? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

648 views

1 answer

hadoop - Will there be Shuffle and sort in Map only task?

Does the shuffle and sort phase come before the end of the map task or does it come after the output is ... written into the final output files. See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

549 views

1 answer

hadoop - MRUnit with Avro NullPointerException in Serialization

I'm trying to test a Hadoop .mapreduce Avro job using MRUnit. I am receiving a NullPointerException as seen below. ... mapDriver.runTest(); } } See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

534 views

1 answer

hadoop - Why do I need to keep hbase/lib folder in hdfs?

I have a main cluster which has some data in Hbase, and I want to replicate it. I've already created a ... Any ideas why it is happening? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

733 views

1 answer

hadoop - Pig: is it possible to write a loop over variables in a list?

I have to loop over 30 variables in a list [var1,var2, ... , var30] and for each variable I use some ... manually 30 times in my code? Thanks! See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

610 views

1 answer

hadoop - How to configure oozie workflow for multi-input path with multiple mappers

Can any one help me in configuring a work flow with Map-Reduce action, that takes multiple input paths each ... this case? Regards, Krish. See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

663 views

1 answer

hadoop - Difference between spark Vectors and scala immutable Vector?

I am writing a project for Spark 1.4 in Scala and am currently in between converting my initial input data into ... than the other? Thank you. See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

773 views

1 answer

hadoop - Flume HDFS sink keeps rolling small files

I'm trying to stream twitter data into hdfs using flume and this: https://github.com/cloudera/cdh-twitter- ... path due to under-replication See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

Categories

Just Browsing Browsing

Most popular tags