Spark shell is great but one of the major issues is the amount of logging it dishes out, it can get frustrating when you are trying to debug things.

Easily solved though.

In your SPARK_HOME/conf directory you’ll find a log4j.properties.template. Make a copy of it.

cp log4j.properties.template log4.properties

Edit log4j.properties with your favourite text editor and change:

log4j.rootCategory=INFO, console

To

log4j.rootCategory=WARN, console

When you restart the Spark shell you’ll have a fighting chance of seeing the output.