AnsweredAssumed Answered

Metrics DB: How to interpret Process_Begin_Time and Process_End_Time

Question asked by dannyman on Feb 24, 2016
Latest reply on May 24, 2016 by maprcommunity

Looking to extend my metrics reporting tool: https://github.com/dannyman/mapr-metrics-report

 

As a start, I'd like to add I what time did a job begin and end. Here's what I get from MySQL:

 

That is not Epoch time ... it is not ... what the heck is this? These are counters, aren't they?

 

It looks like I want the TIME_ fields from the JOB table ... is there anything I can query to get start / end for the Map / Reduce phases?

 

   

mysql> select ATTR_TYPE,ATTR_NAME,ATTR_VALUE from JOB_ATTRIBUTES where JOB_ID like "job_%_081%" and ATTR_NAME like "Process_%_Time";
    +-----------+--------------------+-------------------+
    | ATTR_TYPE | ATTR_NAME          | ATTR_VALUE        |
    +-----------+--------------------+-------------------+
    | Job       | Process_Begin_Time | 17080113956838017 |
    | Job       | Process_End_Time   | 17080114395377537 |
    | Map       | Process_Begin_Time | 16710000950879316 |
    | Map       | Process_End_Time   | 16710001087094847 |
    | Reduce    | Process_Begin_Time | 370113005958701   |
    | Reduce    | Process_End_Time   | 370113308282690   |
    | Job       | Process_Begin_Time | 18380346978690412 |
    | Job       | Process_End_Time   | 18380347621750250 |
    | Map       | Process_Begin_Time | 18195198107986148 |
    | Map       | Process_End_Time   | 18195198527780265 |
    | Reduce    | Process_Begin_Time | 185148870704264   |
    | Reduce    | Process_End_Time   | 185149093969985   |
    | Job       | Process_Begin_Time | 394294044261035   |
    | Job       | Process_End_Time   | 394294051911697   |
    | Map       | Process_Begin_Time | 371100274172168   |
    | Map       | Process_End_Time   | 371100278527898   |
    | Reduce    | Process_Begin_Time | 23193770088867    |
    | Reduce    | Process_End_Time   | 23193773383799    |
    | Job       | Process_Begin_Time | 13237854065414404 |
    | Job       | Process_End_Time   | 13237854550435601 |
    | Map       | Process_Begin_Time | 13052303551638741 |
    | Map       | Process_End_Time   | 13052303861064901 |
    | Reduce    | Process_Begin_Time | 185550513775663   |
    | Reduce    | Process_End_Time   | 185550689370700   |
    | Job       | Process_Begin_Time | 13769904234525974 |
    | Job       | Process_End_Time   | 13769904290209086 |
    | Map       | Process_Begin_Time | 13768454618070426 |
    | Map       | Process_End_Time   | 13768454673739463 |
    | Reduce    | Process_Begin_Time | 1449616455548     |
    | Reduce    | Process_End_Time   | 1449616469623     |
    | Job       | Process_Begin_Time | 13791648876257894 |
    | Job       | Process_End_Time   | 13791648935178724 |
    | Map       | Process_Begin_Time | 13768455010568098 |
    | Map       | Process_End_Time   | 13768455068939498 |
    | Reduce    | Process_Begin_Time | 23193865689796    |
    | Reduce    | Process_End_Time   | 23193866239226    |
    | Job       | Process_Begin_Time | 2899233074819     |
    | Job       | Process_End_Time   | 2899233075235     |
    | Map       | Process_Begin_Time | 2899233074819     |
    | Map       | Process_End_Time   | 2899233075235     |
    | Job       | Process_Begin_Time | 24643484903021    |
    | Job       | Process_End_Time   | 24643484905927    |
    | Map       | Process_Begin_Time | 23193868123300    |
    | Map       | Process_End_Time   | 23193868125672    |
    | Reduce    | Process_Begin_Time | 1449616779721     |
    | Reduce    | Process_End_Time   | 1449616780255     |
    | Job       | Process_Begin_Time | 2899233721739     |
    | Job       | Process_End_Time   | 2899233722669     |
    | Map       | Process_Begin_Time | 2899233721739     |
    | Map       | Process_End_Time   | 2899233722669     |
    | Job       | Process_Begin_Time | 13954013788248562 |
    | Job       | Process_End_Time   | 13954013895284194 |
    | Map       | Process_Begin_Time | 13768462778453704 |
    | Map       | Process_End_Time   | 13768462879263229 |
    | Reduce    | Process_Begin_Time | 185551009794858   |
    | Reduce    | Process_End_Time   | 185551016020965   |
    | Job       | Process_Begin_Time | 12924794430802882 |
    | Job       | Process_End_Time   | 12924794877198681 |
    | Map       | Process_Begin_Time | 12924794430802882 |
    | Map       | Process_End_Time   | 12924794877198681 |
    | Job       | Process_Begin_Time | 13110354636744202 |
    | Job       | Process_End_Time   | 13110354839801176 |
    | Map       | Process_Begin_Time | 12924803313052288 |
    | Map       | Process_End_Time   | 12924803453985802 |
    | Reduce    | Process_Begin_Time | 185551323691914   |
    | Reduce    | Process_End_Time   | 185551385815374   |
    | Job       | Process_Begin_Time | 2901155480762     |
    | Job       | Process_End_Time   | 2901155481744     |
    | Map       | Process_Begin_Time | 2901155480762     |
    | Map       | Process_End_Time   | 2901155481744     |
    | Job       | Process_Begin_Time | 32596179425201383 |
    | Job       | Process_End_Time   | 32596180497555921 |
    | Map       | Process_Begin_Time | 32224248919019161 |
    | Map       | Process_End_Time   | 32224249420048491 |
    | Reduce    | Process_Begin_Time | 371930506182222   |
    | Reduce    | Process_End_Time   | 371931077507430   |
    | Job       | Process_Begin_Time | 873792670203968   |
    | Job       | Process_End_Time   | 873792689531532   |
    | Map       | Process_Begin_Time | 500974415058022   |
    | Map       | Process_End_Time   | 500974432967731   |
    | Reduce    | Process_Begin_Time | 372818255145946   |
    | Reduce    | Process_End_Time   | 372818256563801   |
    | Job       | Process_Begin_Time | 413595186756277   |
    | Job       | Process_End_Time   | 413595187297692   |
    | Map       | Process_Begin_Time | 40776984089601    |
    | Map       | Process_End_Time   | 40776984241924    |
    | Reduce    | Process_Begin_Time | 372818202666676   |
    | Reduce    | Process_End_Time   | 372818203055768   |
    | Job       | Process_Begin_Time | 1118454917347133  |
    | Job       | Process_End_Time   | 1118454921060120  |
    | Map       | Process_Begin_Time | 745636595955046   |
    | Map       | Process_End_Time   | 745636597706583   |
    | Reduce    | Process_Begin_Time | 372818321392087   |
    | Reduce    | Process_End_Time   | 372818323353537   |
    | Job       | Process_Begin_Time | 20663756989327421 |
    | Job       | Process_End_Time   | 20663757537855544 |
    | Map       | Process_Begin_Time | 20290937990740965 |
    | Map       | Process_End_Time   | 20290938170811903 |
    | Reduce    | Process_Begin_Time | 372818998586456   |
    | Reduce    | Process_End_Time   | 372819367043641   |
    | Job       | Process_Begin_Time | 34978128229114837 |
    | Job       | Process_End_Time   | 34978129478757820 |
    | Map       | Process_Begin_Time | 34232485984547349 |
    | Map       | Process_End_Time   | 34232486593599803 |
    | Reduce    | Process_Begin_Time | 745642244567488   |
    | Reduce    | Process_End_Time   | 745642885158017   |
    | Job       | Process_Begin_Time | 14318705772070902 |
    | Job       | Process_End_Time   | 14318706322158115 |
    | Map       | Process_Begin_Time | 14318705772070902 |
    | Map       | Process_End_Time   | 14318706322158115 |
    | Job       | Process_Begin_Time | 14691543100283488 |
    | Job       | Process_End_Time   | 14674067309415507 |
    | Map       | Process_Begin_Time | 14318720222262935 |
    | Map       | Process_End_Time   | 14305613386013442 |
    | Reduce    | Process_Begin_Time | 372822878020553   |
    | Reduce    | Process_End_Time   | 368453923402065   |
    +-----------+--------------------+-------------------+

    116 rows in set (0.67 sec)

Outcomes