Recent questions tagged hadoop

0 votes

469 views

1 answer

hadoop - Is it possible to append to HDFS file from multiple clients in parallel?

Basically whole question is in the title. I'm wondering if it's possible to append to file located on HDFS from ... 't seems to work. Thanks, See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

532 views

1 answer

hadoop - Forward fill missing values in Spark/Python

I am attempting to fill in missing values in my Spark dataframe with the previous non-null value (if it ... Hadoop suite of tools? Thanks! See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

593 views

1 answer

hadoop - How does Hive decide when to use map reduce and when not to?

As a simple example, select * from tablename; DOES NOT kick in map reduce, while select count(*) from tablename; ... use map reduce (by hive)? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

394 views

1 answer

hadoop - Requests hang when using Hiveserver2 Thrift Java client

This is a follow up question to this question where I ask what the Hiveserver 2 thrift java client API is. ... 3 would help narrow my search. See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

643 views

1 answer

hadoop - Finding directories older than N days in HDFS

Can hadoop fs -ls be used to find all directories older than N days (from the current date)? I am trying to ... days prior to the current date. See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

708 views

1 answer

hadoop - How to generate Date Series in HIVE? (Creating table)

Suppose I currently have a table that has 1 row for each account and the data in the tables are: Account Number ... to do that as well? Thanks! See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

710 views

1 answer

hadoop - dfs.namenode.servicerpc-address or dfs.namenode.rpc-address is not configured

I was trying to configure hadoop with one name node and four data nodes. I was able to successfully configure the name ... What do I need to do? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

438 views

1 answer

hadoop - A way to read table data from Mysql to Pig

Everyone know that Pig have supported DBStorage, but they are only supported load results from Pig to mysql like that ... but not success..... See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

591 views

1 answer

hadoop - HDFS file watcher

Can I have file watcher on HDFS? Scenario: The files are landing on HDFS continuously.I want to start a ... help will be appreciated.Thanks. See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

736 views

1 answer

hadoop - get "ERROR: Can't get master address from ZooKeeper; znode data == null" when using Hbase shell

I installed Hadoop2.2.0 and Hbase0.98.0 and here is what I do : $ ./bin/start-hbase.sh $ ./bin ... </value> </property> </configuration> See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

451 views

1 answer

hadoop - Google Analytics - Getting raw data logs

I have an app that sends data to Google Analytics. I am interested in accessing and storing this data ... /thoughts/suggestions are appreciated. See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

868 views

1 answer

hadoop yarn - How to limit the number of retries on Spark job failure?

We are running a Spark job via spark-submit, and I can see that the job will be re-submitted in the ... "GC overhead limit exceeded" issue. See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

434 views

1 answer

hadoop - Spark + Scala transformations, immutability & memory consumption overheads

I have gone through some videos in Youtube regarding Spark architecture. Even though Lazy evaluation, Resilience of data ... cache () works? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

643 views

1 answer

hadoop - yarn is not honouring yarn.nodemanager.resource.cpu-vcores

I am using Hadoop-2.4.0 and my system configs are 24 cores, 96 GB RAM. I am using following configs mapreduce ... . Am I doing something wrong?? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.4k views

1 answer

hadoop - How to load CSV data with enclosed by double quotes and separated by tab into HIVE table?

I am trying to load data from a csv file in which the values are enclosed by double quotes '"' and tab ... and ignore header of the file? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

560 views

1 answer

hadoop - Reading HDFS and local files in Java

I want to read file paths irrespective of whether they are HDFS or local. Currently, I pass the local paths ... other better way than this? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

523 views

1 answer

hadoop - How to know what is the reason for ClosedChannelExceptions with spark-shell in YARN client mode?

I have been trying to run spark-shell in YARN ... scheduler.cluster.YarnSchedulerBackend$YarnSchedulerEndpoint$$anonfun$receiveAndReply$1$$anonfun$applyOrElse$1.apply(YarnScheduler...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

500 views

1 answer

hadoop - Is caching the only advantage of spark over map-reduce?

I have started to learn about Apache Spark and am very impressed by the framework. Although one thing which keeps ... educates me more on spark. See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

573 views

1 answer

hadoop - Spark RDD - is partition(s) always in RAM?

We all know Spark does the computation in memory. I am just curious on followings. If I create 10 RDD in my ... size, where will data to stored? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

500 views

1 answer

hadoop 2.2.0 64-bit installing but cannot start

I am trying to install Hadoop 2.2.0 Cluster on the servers. For now all the servers are 64-bit, I download ... , what do the other errors mean? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

490 views

1 answer

hadoop - SparkSQL vs Hive on Spark - Difference and pros and cons?

SparkSQL CLI internally uses HiveQL and in case Hive on spark(HIVE-7292) , hive uses spark as backend engine. ... and cons of both approaches? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

598 views

1 answer

hadoop - java.net.URISyntaxException when starting HIVE

I am new in HIVE. I have already set up hadoop and it works well, and I want to set up Hive. ... :user.name%7D Are there any solutions? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

534 views

1 answer

hadoop - PIG how to count a number of rows in alias

I did something like this to count the number of rows in an alias in PIG: logs = LOAD 'log' logs_w_one = ... me if there is a better way! See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

744 views

1 answer

hadoop copy a local file system folder to HDFS

I need to copy a folder from local file system to HDFS. I could not find any example of moving a folder( ... /Source-Folder-To-Copy HDFS-URI See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

500 views

1 answer

hadoop - No data nodes are started

I am trying to setup Hadoop version 0.20.203.0 in a pseudo distributed configuration using the following guide: http://www. ... *************/ See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

651 views

1 answer

hadoop - PIG how to count a number of rows in alias

I did something like this to count the number of rows in an alias in PIG: logs = LOAD 'log' logs_w_one = ... me if there is a better way! See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

581 views

1 answer

hadoop - Where does Hive store files in HDFS?

I'd like to know how to find the mapping between Hive tables and the actual HDFS files (or rather, directories ... Hive store its files in HDFS? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

442 views

1 answer

hadoop - Avro vs. Parquet

I'm planning to use one of the hadoop file format for my hadoop related project. I understand parquet is efficient ... to me in simple terms? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

Categories

Just Browsing Browsing

Most popular tags