[PocketFlow]解决在coco上mAP非常低的bug

1.问题

继上次训练挂起的bug后，又遇到了现在评估时AP非常低的bug。具体有多低呢？Pelee论文中提到，用128的batchsize大小在coco数据集上训练70K次迭代后，AP@0.5:0.95为22.4，而我用32的batchsize反复微调之后，最后AP也只从2.9上升到了3.7...下图为训练的过程：

2.解决

其实看loss和accuracy还是可以的，但是ap就是上不去，粗略想到了4个地方可能存在的问题：

训练有误

因为尝试了各种学习率，而且在各个学习率下都是训练到ap和loss都不变之后才改变学习率，因此我想不出训练还有什么其他花样了...排除
数据有误

因为coco的数据是自己转换的，说实话对这块还是有点不放心的。况且之前训练的bug已经检查过数据了，看样子是没什么问题。现在再检查工作量也比较大，存疑，先放着
模型有误

模型在voc数据集上能成功复现，排除
计算有误

因为coco评估的脚本是在voc上改的，生成json文件之后再使用官方的cocoapi计算。所以有很大可能是生成json文件的脚本哪里写错了

综上，先排查评估是否有计算错误。

但是没看出来...于是，我寻思着在原来的voc_eval.py文件上做一些修改，以适配coco，再用voc的方式评估。虽然计算方式有差别，但不会差太远。如果ap变化不大，那说明计算方式没有问题，需要检查数据(最怕的就是这种情况，因为工作量很大，而且数据也容易出错，还好gt标错的坑已经提前踩了，想想就可怕)；反之就是评估计算有问题。

# voc_eval.py 程序结构

def do_python_eval(dataset_path, pred_path, use_07=True):

aps = []

#对每个类别：执行

rec, prec, ap = voc_eval(filename, # 每个类别的预测文件result_x.txt

    os.path.join(dataset_path, anno_files), # 每张图片的标注文件

    os.path.join(dataset_path, all_images_file), # 所有图片的文件名文件

    cls_name, # 类别名

    cache_path, # 用于暂存所有图片的标注

    ovthresh=0.5,

    use_07_metric=use_07_metric)

aps += [ap]

#将rec, prec和ap 存到对应类别的 xx_pr.pkl文件

#打印AP和mAP

def voc_eval(detpath,

                     annopath,

                     imagesetfile,

             	     classname,

                     cachedir,

                     ovthresh=0.5,

                     use_07_metric=True):

    #1.从imagesetfile中读取所有图片文件名

    #2.如果cachedir中的annots.pkl文件不存在，则将标注按文件名打包之后写入annots.pkl；否则加载文件到recs字典中（key为文件名）

    #3.提取该类别的标注，按文件名打包存到class_recs字典中

    #4.从txt文件读取该类别的标注

    #5.计算rec, prec, 用voc_ap计算ap

    #6.返回rec, prec, ap

#如果要更改以适配COCO，需要：

#获取所有image_id

#从minival.json中按image_id提取标注

分析了一波之后改好了代码，也能正常运行（非常慢）。要命的是好不容易把文件都读完了，结果报错了。显示keyerror：

{213035: [{'name': 'scissors', 'bbox': [314.25, 168.05, 79.57, 53.75]}, {'name': 'scissors', 'bbox': [238.06, 170.62, 89.75, 64.11]}, {'name': 'person', 'bbox': [0.0, 110.12, 311.59, 235.62]}, {'name': 'person', 'bbox': [195.75, 2.88, 260.04, 109.39]}, {'name': 'bowl', 'bbox': [177.04, 0.0, 66.69, 135.89]}, {'name': 'person', 'bbox': [305.0, 53.24, 330.51, 373.76]}]}

r = [obj for obj in dic[213035] if obj['name'] == 'person']

# 大致是说obj没有name属性

输出改obj的iamge_id之后到minival.json去找，发现这个标注的类别是82还是83...coco不是80个类吗？给我整懵逼了...

但是，突然灵光一闪，我似乎已经找到问题所在了。coco数据集虽然有80个类，但是却不是顺序排下来的，中间有跳过的序号，所以真实的序号是从1到90，这个项目之前做过一个转换：

labelmap = {

    "none_of_the_above": 0,

    "1": 1,

    "2": 2,

    "3": 3,

    "4": 4,

    "5": 5,

    "6": 6,

    "7": 7,

    "8": 8,

    "9": 9,

    "10": 10,

    "11": 11,

    "13": 12,

    "14": 13,

    "15": 14,

    "16": 15,

    "17": 16,

    "18": 17,

    "19": 18,

    "20": 19,

    "21": 20,

    "22": 21,

    "23": 22,

    "24": 23,

    "25": 24,

    "27": 25,

    "28": 26,

    "31": 27,

    "32": 28,

    "33": 29,

    "34": 30,

    "35": 31,

    "36": 32,

    "37": 33,

    "38": 34,

    "39": 35,

    "40": 36,

    "41": 37,

    "42": 38,

    "43": 39,

    "44": 40,

    "46": 41,

    "47": 42,

    "48": 43,

    "49": 44,

    "50": 45,

    "51": 46,

    "52": 47,

    "53": 48,

    "54": 49,

    "55": 50,

    "56": 51,

    "57": 52,

    "58": 53,

    "59": 54,

    "60": 55,

    "61": 56,

    "62": 57,

    "63": 58,

    "64": 59,

    "65": 60,

    "67": 61,

    "70": 62,

    "72": 63,

    "73": 64,

    "74": 65,

    "75": 66,

    "76": 67,

    "77": 68,

    "78": 69,

    "79": 70,

    "80": 71,

    "81": 72,

    "82": 73,

    "84": 74,

    "85": 75,

    "86": 76,

    "87": 77,

    "88": 78,

    "89": 79,

    "90": 80

}

COCO_LABELS = {

    "bench": (14, 'outdoor'),

    "skateboard": (37, 'sports'),

    "toothbrush": (80, 'indoor'),

    "person": (1, 'person'),

    "donut": (55, 'food'),

    "none": (0, 'background'),

    "refrigerator": (73, 'appliance'),

    "horse": (18, 'animal'),

    "elephant": (21, 'animal'),

    "book": (74, 'indoor'),

    "car": (3, 'vehicle'),

    "keyboard": (67, 'electronic'),

    "cow": (20, 'animal'),

    "microwave": (69, 'appliance'),

    "traffic light": (10, 'outdoor'),

    "tie": (28, 'accessory'),

    "dining table": (61, 'furniture'),

    "toaster": (71, 'appliance'),

    "baseball glove": (36, 'sports'),

    "giraffe": (24, 'animal'),

    "cake": (56, 'food'),

    "handbag": (27, 'accessory'),

    "scissors": (77, 'indoor'),

    "bowl": (46, 'kitchen'),

    "couch": (58, 'furniture'),

    "chair": (57, 'furniture'),

    "boat": (9, 'vehicle'),

    "hair drier": (79, 'indoor'),

    "airplane": (5, 'vehicle'),

    "pizza": (54, 'food'),

    "backpack": (25, 'accessory'),

    "kite": (34, 'sports'),

    "sheep": (19, 'animal'),

    "umbrella": (26, 'accessory'),

    "stop sign": (12, 'outdoor'),

    "truck": (8, 'vehicle'),

    "skis": (31, 'sports'),

    "sandwich": (49, 'food'),

    "broccoli": (51, 'food'),

    "wine glass": (41, 'kitchen'),

    "surfboard": (38, 'sports'),

    "sports ball": (33, 'sports'),

    "cell phone": (68, 'electronic'),

    "dog": (17, 'animal'),

    "bed": (60, 'furniture'),

    "toilet": (62, 'furniture'),

    "fire hydrant": (11, 'outdoor'),

    "oven": (70, 'appliance'),

    "zebra": (23, 'animal'),

    "tv": (63, 'electronic'),

    "potted plant": (59, 'furniture'),

    "parking meter": (13, 'outdoor'),

    "spoon": (45, 'kitchen'),

    "bus": (6, 'vehicle'),

    "laptop": (64, 'electronic'),

    "cup": (42, 'kitchen'),

    "bird": (15, 'animal'),

    "sink": (72, 'appliance'),

    "remote": (66, 'electronic'),

    "bicycle": (2, 'vehicle'),

    "tennis racket": (39, 'sports'),

    "baseball bat": (35, 'sports'),

    "cat": (16, 'animal'),

    "fork": (43, 'kitchen'),

    "suitcase": (29, 'accessory'),

    "snowboard": (32, 'sports'),

    "clock": (75, 'indoor'),

    "apple": (48, 'food'),

    "mouse": (65, 'electronic'),

    "bottle": (40, 'kitchen'),

    "frisbee": (30, 'sports'),

    "carrot": (52, 'food'),

    "bear": (22, 'animal'),

    "hot dog": (53, 'food'),

    "teddy bear": (78, 'indoor'),

    "knife": (44, 'kitchen'),

    "train": (7, 'vehicle'),

    "vase": (76, 'indoor'),

    "banana": (47, 'food'),

    "motorcycle": (4, 'vehicle'),

    "orange": (50, 'food')

}

妈的生成json文件的时候我忘了换回来了（其实想一下好像不这样来回转也行）...所以只有序号从1到11的类别能够对上。简单修改后，得到真正的预测结果:

 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.199

 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.343

 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.201

 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.030

 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.200

 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.365

 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.201

 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.295

 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.314

 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.054

 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.342

 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.548

与论文中的22.4和38.3相比还有差距，但是至少没有开始吓人了，再根据正确的AP微调一下应该还能提高点。

算是误打误撞解决了？

虽然这次的bug也调了一星期，但是明显没有上次那么慌了。

巴特西

[PocketFlow]解决在coco上mAP非常低的bug

1.问题

2.解决

最新文章

热门文章