开发者

Scala Java interop typechecking

开发者 https://www.devze.com 2023-03-14 03:24 出处:网络
I am unable to fix these typechecking errors in Scala: package junk import org.apache.hadoop.conf.Configuration

I am unable to fix these typechecking errors in Scala:

package junk

import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.io.IntWritable
import org.apache.hadoop.io.Text
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat
import org.apache.hadoop.mapreduce.Job

object Sample {
  val conf: Configuration = new Configuration()
  val job: Job = new Job(conf, "sample Hadoop MapReduce program");

  job.setInputFormatClass(classOf[TextInputFormat])
  job.setOutputFormatClass(classOf[TextOutputFormat])
  job.setOutputFormatClass(classOf[TextOutputFormat[Text, IntWritable]])
}    

The error messages are

Sample.scala:14: error: type mismatch;
found   : java.lang.Class[org.apache.hadoop.mapreduce.lib.input.TextInputFormat](classOf[org.apache.hadoop.mapreduce.lib.input.TextInputFormat])
required: java.lang.Class[_ <: org.apache.hadoop.mapreduce.InputFormat]
 job.setInputFormatClass(classOf[TextInputFormat])
                                ^
Sample.scala:15: error: class TextOutputFormat takes type parameters
 job.setOutputFormatClass(classOf[TextOutputFormat])
                                  ^
Sample.scala:16: error: type mismatch;
found   : java.lang.Class[org.apache.hadoop.mapreduce.lib.output.TextOutputFormat[org.apache.hadoop.io.Text,org.apache.hadoop.io.IntWritable]](classOf[org.apache.hadoop.mapreduce.lib.output.TextOutput开发者_运维百科Format])
required: java.lang.Class[_ <: org.apache.hadoop.mapreduce.OutputFormat]
 job.setOutputFormatClass(classOf[TextOutputFormat[Text, IntWritable]])

Note that I have tried to setOutputFormatClass in two different ways and both fail.

I am trying to write Hadoop WordCount: http://wiki.apache.org/hadoop/WordCount in Scala. The equivalent lines in Java are

job.setInputFormatClass(TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);

API docs for Job: http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/Job.html


See Programming In Scala, Existential Types.

val textInputFormatClass = classOf[TextInputFormat].
     asInstanceOf[Class[T] forSome {type T <: InputFormat[String, String]}]

job.setInputFormatClass(textInputFormatClass)

Substitute the types that you are using for [String, String]

scala> val textInputFormatClass = classOf[TextInputFormat].asInstanceOf[Class[T] forSome {type T <: InputFormat[String, String]}]
textInputFormatClass: java.lang.Class[_ <: org.apache.hadoop.mapreduce.InputFormat[String,String]] = class org.apache.hadoop.mapreduce.lib.input.TextInputFormat

scala> val job = new Job
job: org.apache.hadoop.mapreduce.Job = org.apache.hadoop.mapreduce.Job@4a205144

scala> job.setInputFormatClass(textInputFormatClass)

scala>


As of Scala 2.9.1 (not Scala 2.9.0 -- see https://issues.scala-lang.org/browse/SI-4603), this also works:

job.setInputFormatClass(classOf[TextInputFormat])
0

精彩评论

暂无评论...
验证码 换一张
取 消

关注公众号