How to enable debug logging for partition pruning phase in Drill

Document created by Hao Zhu Employee on Feb 7, 2016
Version 1Show Document
  • View in full screen mode

Author: Hao Zhu

 

Original Publication Date: September 11, 2015

 

Environment :

Drill 1.1

Goal:

How to enable debug logging for partition pruning phase in Drill.

This can help troubleshoot when the partition pruning is not happening due to some reason.

Solution:

1. Add below lines in logback.xml on all drill nodes.

<logger name="org.apache.drill.exec.planner.logical.partition.PruneScanRule" additivity="false"> <level value="debug" /> <appender-ref ref="FILE" /> </logger>

2. Restart all drillbits.
3. Run a query which should utilize partition pruning.

For example:

> select * from dfs.`/drill/part1` where dir0='2015' and dir1='01'; +----------------+-------+-------+ |    columns     | dir0  | dir1  | +----------------+-------+-------+ | ["1","'111'"]  | 2015  | 01    | +----------------+-------+-------+ 1 row selected (0.219 seconds)

Check the foreman drillbit.log, and below debug information should show:

:
DEBUG o.a.d.e.p.l.partition.PruneScanRule - Attempting to prune AND(=($1, '2015'), =($2, '01')) 
DEBUG o.a.d.e.p.l.partition.PruneScanRule - Pruned [maprfs:/drill/part1/2015/01/01.csv, maprfs:/drill/part1/2015/02/02.csv, maprfs:/drill/part1/2015/03/03.csv, maprfs:/drill/part1/2014/11/11.csv, maprfs:/drill/part1/2014/12/12.csv] => [maprfs:/drill/part1/2015/01/01.csv]

Attachments

    Outcomes