Cutting down on the verbosity of #Spark messages

Spark shell is great but one of the major issues is the amount of logging it dishes out, it can get frustrating when you are trying to debug things.

Easily solved though.

In your SPARK_HOME/conf directory you’ll find a Make a copy of it.


Edit with your favourite text editor and change:

log4j.rootCategory=INFO, console


log4j.rootCategory=WARN, console

When you restart the Spark shell you’ll have a fighting chance of seeing the output.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: