开发者

Hadoop java mapper job executing on slave node, directory issue

开发者 https://www.devze.com 2023-04-07 11:58 出处:网络
As part of my Java mapper I have a command executes some standalone code on a local slave node.When I run a code it executes fine, unless it is trying to access some local files in which case I get th

As part of my Java mapper I have a command executes some standalone code on a local slave node. When I run a code it executes fine, unless it is trying to access some local files in which case I get the error that it cannot locate those files.

Digging a little deeper i开发者_开发技巧t seems to be executing from the following directory:

/data/hadoop/mapred/local/taskTracker/{user}/jobcache/job_201109261253_0023/attempt_201109261253_0023_m_000001_0/work

But I am intending to execute from a local directory where the relevant files are located:

/home/users/{user}/input/jobname

Is there a way in java/hadoop to force the execution from the local directory, instead of the jobcache directory automatically created in hadoop?

Is there perhaps a better way to go about this?

Any help on this would be greatly appreciated!


A workaround method I'm using right now that works consists of copying all the relevant files over to the jobcache working directory. Then you can copy the results back to user directory if necessary.

Unfortunately this doesn't fully answer the question, but hopefully provides a useful workaround for others.

Cheers,

Joris

0

精彩评论

暂无评论...
验证码 换一张
取 消