Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Single definition point for Hadoop properties #271

Open
simleo opened this issue Feb 8, 2018 · 0 comments
Open

Single definition point for Hadoop properties #271

simleo opened this issue Feb 8, 2018 · 0 comments

Comments

@simleo
Copy link
Member

simleo commented Feb 8, 2018

We have several explicit, duplicate references to Hadoop property keys, things like conf["mapreduce.job.reduces"] = "2". We should define them once and for all in some common module and use the corresponding variables everywhere else.

For instance:

NUM_REDUCES = "mapreduce.job.reduces"
[...]

[pydoop/prop.py]

And then:

conf = {}
import pydoop.prop as prop
conf[prop.NUM_REDUCES] = "2"

To simplify the process, we can use the same names as in Hadoop itself. In this case (from MRJobConfig.java):

public static final String NUM_REDUCES = "mapreduce.job.reduces";
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant