AnsweredAssumed Answered

How to fix stale containers ?

Question asked by rkumar_xactly on Feb 1, 2017
Latest reply on Mar 12, 2017 by mufeed
Hi All,
I see so many stale containers in our cluster. How do I fix them ?
pdsh> /opt/mapr/server/mrconfig info containers resync local
hdn11: Stale containers: 2754 3009 7059 7073 10011 15938 20091 21126 22656 23349 23352 24067 40232 41691 41719 43090 43157 45581 46982 46986 47014 47253 47401 47896 48207 48266 48891 49176 49328 49567 49717 49797 49889 50465 50575 50617 50677 50744 50983 51424 51617 51647 51655 51670 51730 51863 51864 51940 52084 52224 52293 52430 52593 52636 52762 52832 52935 52940 52949 53355 53425 53531 53598 53599 53664 53708 53891 53892 53893 53894 53895 53896 53897 53899 53901 53903 53906 53907 53908 53909 53910 53911 53912 53913 53914 53915 53917 53921 53923 53925 53927 53929 53931 53933 53935 53937 53939 53940 53942 53945
hdn12: Containers in resync: 51955
hdn12: Stale containers: 47178 51955
hdn13: Containers in resync: 50093 50888 51006 53053
hdn13: Stale containers: 50093 50888 51006 53053
hdn7: Stale containers: 2734 3049 3268 3278 5739 6065 6411 6518 7019 7240 7761 8037 8126 8784 9101 9602 9730 9748 9948 9962 9975 10027 10090 17026 19035 19238 19287 19293 19398 19488 19594 19595 19648 20076 20116 21476 22458 22662 22774 23630 23696 23784 24081 24128 24145 24202 25136 26182 26371 26497 26919 27023 27033 27185 27244 27249 34493 35156 35158 35255 35454 35692 36001 37251 37252 37324 37508 37549 37558 37559 37573 37575 37866 37963
hdn1: Stale containers: 2394 2718 2723 2730 2732 2734 2743 3014 3050 3191 3192 3268 3276 3278 3304 5718 5725 6031 6354 6384 6419 6452 6493 6501 6518 6743 6938 6945 7026 7597 7608 7686 7705 7761 7839 7867 7874 7887 8084 8120 8121 8126 8132 8330 8346 8397 8398 8834 9219 9538 9730 9746 9925 9926 9942 9947 9958 9961 9969 9975 10026 10027 10044 10052 10058 10074 10084 10091 15106 15939 16895 19031 19035 19039 19238 19488 19633 19637 19648 19651 19701 19710 20055 20064 20072 20075 20087 20114 20118 20129 20960 21466 21476 21486 22450 22451 22458 22658 22783 23089 23097 23102 23492 23631 24091 24118 24145 24775 24811 25109 25224 25233 26066 26158 26168 26893 26915 26919 27010 27030 27118 27185 27246 27248 35114 35156 35255 35359 35998 36001 37251 37252 37253 37362 37366 37490 37575 37798 37800 37803 37958 37963 37964
hdn2: Stale containers: 2678 2717 2748 3021 3050 3192 3264 6379 6411 6419 6743 6966 7063 7538 7597 7608 7705 7709 7734 7803 7867 7874 7879 7887 8327 8398 9667 9748 9934 9938 9962 17026 19036 19038 19039 19237 19277 19283 19287 19566 19648 19651 19871 19886 20060 20082 20087 20095 20127 20749 21486 22776 22822 22826 23097 24119 24145 24383 24598 25224 25233 26066 26148 26149 26164 26248 26371 26893 26950 26959 27009 27020 27021 27104 27185 34493 34507 35163 35239 35719 36000 36155 37038 37549 37783 37787 37803 37962
hdn4: Stale containers: 2389 2726 2737 2740 2753 3045 3191 3264 3268 5726 6070 6340 6349 6424 6938 6945 7729 7874 8105 8118 8128 8327 9094 9238 9422 9971 10014 10021 10061 10087 18665 19031 19230 19232 19235 19597 19633 19704 19863 20097 20255 20962 20997 21586 22481 22780 23087 23630 23726 23779 23783 24081 24096 24128 24233 24397 24773 25135 27030 27108 27180 34517 35158 35165 35259 35454 35465 36104 37362 37455
hdn9: Stale containers: 2717 2724 3056 3189 3276 3304 3313 5739 6379 6452 6500 6966 7026 7602 7608 7692 7724 7734 7735 7744 7748 7761 7777 7803 7838 7872 8121 9732 9943 9949 9975 10004 10084 10087 16865 16894 17026 19283 19287 19399 19704 19705 19871 19886 20077 20107 20108 20111 22477 22776 23102 24081 24096 24202 24811 26148 26168 26248 26387 26497 26940 26950 26959 27033 27104 27191 34493 34517 35156 35158 35239 35255 35359 35454 35675 35692 35702 35705 35717 35719 35736 35996 36354 36357 37038 37252 37358 37455 37573 37787 37800 37829 37834 37866
hdn8: Stale containers: 2678 2724 2842 3010 3014 3024 3038 3040 3056 5725 6349 6515 6519 6743 7014 7063 7538 7631 7686 7692 7705 7722 7881 8084 8346 9094 9101 9948 9949 9973 9976 9983 10021 10026 10027 10060 15939 16865 19232 19277 19398 19568 19594 19637 19701 19886 20056 20067 20072 20077 20097 20113 20116 21466 21586 22481 22486 22662 22774 22780 23785 24118 24129 24145 24233 24811 26148 26164 26182 26371 26875 27010 27021 27030 27063 27104 27118 27163 27180 27185 27206 27244 27246 34511 37362 37363 37787 37798 37800 37963 37964
hdn10: Stale containers: 2678 3012 3189 3192 3264 3268 3278 3313 5712 5739 6387 6411 6452 6493 6500 6515 6695 7014 7019 7602 7608 7686 7692 7724 7734 7735 7748 7777 7803 7838 7872 7879 7881 7887 8025 8037 8117 8118 8123 8129 8130 8327 8397 8398 8401 8784 9602 9925 9971 10053 10061 10071 16865 19287 19293 19398 19399 19488 19568 19579 19633 19637 19648 19705 19863 19886 20060 20064 20067 20100 20108 20749 20960 20997 22458 22477 22662 22774 23696 23784 23791 24096 24119 24811 26148 26164 26168 26371 26875 26940 26950 26959 27009 27010 27021 27030 27033 27063 27104 27118 27156 27163 27185 27206 27244 27246 27248 34511 34517 34518 35158 35255 35454 35464 35692 35702 35717 35719 35998 37038 37065 37322 37324 37452 37455 37460 37549 37798 37847 37856 38118
hdn5: Stale containers: 2842 6354 6424 6966 7538 7665 8084 8111 8123 9732 9969 12935 19594 19698 24091 24233 24773 24811 25136 26371 26834 27163 37559 37856
hdn3: Stale containers: 2723 2732 2734 6325 6945 6966 7014 7063 7538 7597 7879 8037 8123 9421 9603 9667 9925 9931 10045 10053 10058 10071 10076 10084 12935 19036 19038 19232 19293 19579 19698 19710 20072 20076 20086 20107 20127 22774 23492 24081 24091 24119 24128 24129 24233 25109 25136 27023 27118 27180 27191 27206 27248 34511 36001 36155 37065 37559 37847 37964
pdsh>
==========================================
Here is the maprcli dump resync output:
maprcli dump containers -type resync -json
{
"timestamp":1485966665680,
"timeofday":"2017-02-01 08:31:05.680 GMT-0800",
"status":"OK",
"total":5,
"data":[
{
"InstanceCount":1,
"ContainerId":52789,
"Epoch":5,
"Master":"10.250.70.112:5660-10.250.71.112:5660-10.250.72.112:5660--5-VALID",
"ActiveServers":{
"IP:Port":[
"10.250.70.112:5660-10.250.71.112:5660-10.250.72.112:5660--5-VALID",
"10.250.70.110:5660-10.250.71.110:5660-10.250.72.110:5660--5-VALID",
"10.250.71.116:5660-10.250.72.116:5660-10.250.70.116:5660--5-VALID",
"10.250.70.121:5660-10.250.71.121:5660-10.250.72.121:5660--2-RESYNC"
]
},
"InactiveServers":{

},
"UnusedServers":{

},
"OwnedSizeMB":"0 MB",
"SharedSizeMB":"0 MB",
"LogicalSizeMB":"0 MB",
"TotalSizeMB":"0 MB",
"NameContainer":"false",
"CreatorContainerId":0,
"CreatorVolumeUuid":"",
"UseActualCreatorId":true
},
{
"InstanceCount":1,
"ContainerId":53061,
"Epoch":4,
"Master":"10.250.70.112:5660-10.250.71.112:5660-10.250.72.112:5660--4-VALID",
"ActiveServers":{
"IP:Port":[
"10.250.70.112:5660-10.250.71.112:5660-10.250.72.112:5660--4-VALID",
"10.250.70.115:5660-10.250.71.115:5660-10.250.72.115:5660--4-VALID",
"10.250.70.113:5660-10.250.71.113:5660-10.250.72.113:5660--4-VALID",
"10.250.70.122:5660-10.250.71.122:5660-10.250.72.122:5660--2-RESYNC"
]
},
"InactiveServers":{

},
"UnusedServers":{

},
"OwnedSizeMB":"0 MB",
"SharedSizeMB":"0 MB",
"LogicalSizeMB":"0 MB",
"TotalSizeMB":"0 MB",
"NameContainer":"false",
"CreatorContainerId":0,
"CreatorVolumeUuid":"",
"UseActualCreatorId":true
},
{
"InstanceCount":1,
"ContainerId":50062,
"Epoch":7,
"Master":"10.250.70.113:5660-10.250.71.113:5660-10.250.72.113:5660--7-VALID",
"ActiveServers":{
"IP:Port":[
"10.250.70.113:5660-10.250.71.113:5660-10.250.72.113:5660--7-VALID",
"10.250.70.110:5660-10.250.71.110:5660-10.250.72.110:5660--7-VALID",
"10.250.70.117:5660-10.250.71.117:5660-10.250.72.117:5660--7-VALID",
"10.250.70.122:5660-10.250.71.122:5660-10.250.72.122:5660--2-RESYNC"
]
},
"InactiveServers":{

},
"UnusedServers":{

},
"OwnedSizeMB":"0 MB",
"SharedSizeMB":"0 MB",
"LogicalSizeMB":"0 MB",
"TotalSizeMB":"0 MB",
"NameContainer":"false",
"CreatorContainerId":0,
"CreatorVolumeUuid":"",
"UseActualCreatorId":false
},
{
"InstanceCount":1,
"ContainerId":51939,
"Epoch":4,
"Master":"10.250.71.111:5660-10.250.72.111:5660-10.250.70.111:5660--4-VALID",
"ActiveServers":{
"IP:Port":[
"10.250.71.111:5660-10.250.72.111:5660-10.250.70.111:5660--4-VALID",
"10.250.70.118:5660-10.250.71.118:5660-10.250.72.118:5660--4-VALID",
"10.250.70.115:5660-10.250.71.115:5660-10.250.72.115:5660--4-VALID",
"10.250.70.121:5660-10.250.71.121:5660-10.250.72.121:5660--2-RESYNC"
]
},
"InactiveServers":{

},
"UnusedServers":{

},
"OwnedSizeMB":"0 MB",
"SharedSizeMB":"0 MB",
"LogicalSizeMB":"0 MB",
"TotalSizeMB":"0 MB",
"NameContainer":"false",
"CreatorContainerId":0,
"CreatorVolumeUuid":"",
"UseActualCreatorId":true
},
{
"InstanceCount":1,
"ContainerId":6395,
"Epoch":41,
"Master":"10.250.70.110:5660-10.250.71.110:5660-10.250.72.110:5660--41-VALID",
"ActiveServers":{
"IP:Port":[
"10.250.70.110:5660-10.250.71.110:5660-10.250.72.110:5660--41-VALID",
"10.250.70.117:5660-10.250.71.117:5660-10.250.72.117:5660--41-VALID",
"10.250.70.115:5660-10.250.71.115:5660-10.250.72.115:5660--41-VALID",
"10.250.70.122:5660-10.250.71.122:5660-10.250.72.122:5660--2-RESYNC"
]
},
"InactiveServers":{

},
"UnusedServers":{

},
"OwnedSizeMB":"0 MB",
"SharedSizeMB":"0 MB",
"LogicalSizeMB":"0 MB",
"TotalSizeMB":"0 MB",
"NameContainer":"false",
"CreatorContainerId":0,
"CreatorVolumeUuid":"",
"UseActualCreatorId":false
}
]
}

Outcomes