How to Read a Text File Using Spark Scala with Example

Reading a text file in Spark Scala is straightforward. You can use the textFile method.

import org.apache.spark.sql.SparkSession

object ReadTextFile {
    def main(args: Array[String]): Unit = {
      val sparkSession = SparkSession
        .builder()
        .appName("Our First scala spark code")
        .master("local")
        .getOrCreate()
      val salesDF = sparkSession.read.textFile("data/text/data.txt")
      salesDF.show(false)
      sparkSession.stop()
    }
  }
  

Output:

Below list contains some most commonly used options while reading a csv file

• "lineSep": can use 'lineSep' option to define the line separator.
• wholetext: can also use 'wholetext' option to read each input file as a single row..
• compression: Compression codec to use when saving to file.