I'm looking to have a persistent spark context where I can load data frames and from parquet files and query them interactively.
I've heard of using a thrift server for this, but I'm a little unclear how it works.
I see a small amount of information in the Spark SQL and DataFrames - Spark 2.1.1 Documentation, but it's not overly helpful. Equally, I see a few blurbs on the IBM site etc. that don't cover much.
I don't see anything about how to actually load up data frames from parquet, cache them, and query them though (or even anything remotely that complex).
Does anyone know of a tutorial or demo that walks through this, or a place where the information can be learned? Or can someone explain it here to get me started?