MATLAB Answers

Jingyu Ru
0

How to solve" The user home directory '/homes/' for the cluster either did not exist or was not writable by the HADOOP user."

Jingyu Ru
さんによって質問されました 2015 年 7 月 26 日
最新アクティビティ lov kumar
さんによって コメントされました 2019 年 6 月 6 日
Hadoop version 1.2.1
Matlab version 2015a
Linux ubuntu 14.
I install Hadoop for a pseudo-distributed configuration (all of the Hadoop daemons run on a single host).
It is success to run the example 'wordcount'in Hadoop.
And it is success to read the data from the HDFS through the Matlab.
But when I try to run the example in Matlab 'Run mapreduce on a Hadoop Cluster',I failed.
It shows that the Map 0% and Reduce 0%.
There is my Matlab codes
setenv('HADOOP_HOME','/home/rjy/soft/hadoop-1.2.1');
cluster = parallel.cluster.Hadoop;
cluster.HadoopProperties('mapred.job.tracker') = 'localhost:50031';
cluster.HadoopProperties('fs.default.name') = 'hdfs://localhost:8020';
% ds = datastore('hdfs:/user/root/rrr')
outputFolder = '/home/rjy/logs/hadooplog';
mr = mapreducer(cluster);
ds = datastore('airlinesmall.csv','TreatAsMissing','NA',...
'SelectedVariableNames','ArrDelay','ReadSize',1000);
preview(ds)
meanDelay = mapreduce(ds,@meanArrivalDelayMapper,@meanArrivalDelayReducer,mr,...
'OutputFolder',outputFolder)
Error log is shown as follows,
Error using mapreduce (line 100)
The HADOOP job failed to complete.
Error in run_mapreduce_on_a_hadoop (line 12)
meanDelay = mapreduce(ds,@meanArrivalDelayMapper,@meanArrivalDelayReducer,mr,...
Caused by:
The HADOOP job was not able to start MATLAB for attempt 1 of 'MAP' task 0. The user
home directory '/homes/' for the cluster either did not exist or was not writable by
the HADOOP user. Check the documentation on how to set the user home directory for the
cluster.
What's the '/homes/' means? I have never seen it before. My hadoop directory is './home/rjy/soft/hadoop-1.2.1'
I wish to know how to solve this problem. I have tried many methods to solve it.
Please give me some suggestions. Thanks.

  0 件のコメント

サインイン to comment.

1 件の回答

回答者: Rick Amos
2015 年 7 月 27 日
 採用された回答

The '/homes/' folder is related to the Hadoop property 'mapreduce.admin.user.home.dir'. This is used by Hadoop to set the HOME environment variable for worker processes.
The HOME environment variable is used by MATLAB for things such as preferences and caches. This is why as part of setting up Hadoop alongside MATLAB Distributed Computing Server, you need to ensure this property is set to a local folder that is writable by the cluster, for example "/home/hduser".
All required configuration steps are listed at:
I hope this helps,
Rick

  2 件のコメント

Jingyu Ru
2015 年 7 月 28 日
Thanks a lot, I have solved the problem by set the mapred-site.xml,
property name>mapreduce.admin.user.home.dir</name value/home/rjy/soft/hadoop-1.2.1</value> /property /configuration
Thanks again.
lov kumar
2019 年 6 月 6 日
How to fix this issue:
Error using mapreduce (line 125)
The HADOOP job failed to submit. It is possible that there is some issue with the HADOOP configuration.
setenv('HADOOP_PREFIX','C:/hadoop-2.8.0');
getenv('HADOOP_PREFIX');
ds = datastore('hdfs://localhost:9000/lov/airlinesmall.csv',...
'TreatAsMissing','NA',...
'SelectedVariableNames',{'UniqueCarrier','ArrDelay'});
cluster = parallel.cluster.Hadoop;
cluster.HadoopProperties('mapred.job.tracker') = 'localhost:9000';
cluster.HadoopProperties('fs.default.name') = 'hdfs://localhost:8088';
disp(cluster);
mapreducer(cluster);
result = mapreduce(ds,@maxArrivalDelayMapper,@maxArrivalDelayReducer,'OutputFolder','hdfs://localhost:9000/tmp');

サインイン to comment.



Translated by