a +ca7@sfdZddlZddlZddlZddlZddlZddlZddlmZddl Z ddl Z ddl Z ddlZddlmZddlmZddlmZddlmZddlZddlmZee jedksJeejedksJd_ddZGd d d ej Z!d d Z"d`ddZ#daddZ$dbddZ%ddZ&ddZ'Gdddej(Z)ddZ*Gdddej(Z+d d!Z,d"d#Z-Gd$d%d%ej(Z.d&d'Z/Gd(d)d)ej(Z0d*d+Z1d,d-Z2dcd/d0Z3ddd1d2Z4d3d4Z5d5d6Z6d7d8Z7d9d:Z8d;d<Z9d=d>Z:ded?d@Z;dAdBZdfdHdIZ?GdJdKdKZ@dLdMZAdNdOZBdPdQZCdRdSZDdTdUZEdgdWdXZFdYdZZGd[d\ZHd]d^ZIdS)haH Mask R-CNN The main Mask R-CNN model implementation. Copyright (c) 2017 Matterport, Inc. Licensed under the MIT License (see LICENSE for details) Written by Waleed Abdulla Modified by Ondrej Pesek Written error handling in load_image_gt and data_generator Changed the verbosity to correspond GRASS verbosity in detect N) OrderedDict) LooseVersionz1.3z2.0.8cCsr|durf|d}|dt|j7}|jrF|d||7}n|ddd7}|d|j7}t|dS)zxPrints a text message. And, optionally, if a Numpy array is provided it prints it's shape, min, and max values. Nzshape: {:20} zmin: {:10.5f} max: {:10.5f}zmin: {:10} max: {:10}z {}) ljustformatstrshapesizeminmaxdtypeprint)textarrayrUC:/Users/landamar/grass_packager/grass786/addons/i.ann.maskrcnn/etc/maskrcnn/model.pylog+s rcs"eZdZdZdfdd ZZS) BatchNormaExtends the Keras BatchNormalization class to allow a central place to make changes if needed. Batch normalization has a negative effect on training if batches are small so this layer is often frozen (via setting in Config class) and functions as linear layer. Ncst|j|j||dS)a Note about training values: None: Train BN layers. This is the normal mode False: Freeze BN layers. Good when batch size is small True: (don't use). Set layer in training mode even when making inferences training)super __class__call)selfinputsrrrrrCszBatchNorm.call)N)__name__ __module__ __qualname____doc__r __classcell__rrrrr:srcs<t|jr|S|jdvs"Jtfdd|jDS)zComputes the width and height of each stage of the backbone network. Returns: [N, (height, width)]. Where N is the number of stages Zresnet50Z resnet101cs8g|]0}ttd|ttd|gqSr)intmathZceil).0Zstride image_shaperr Ysz+compute_backbone_shapes..)callableBACKBONEZCOMPUTE_BACKBONE_SHAPEnprBACKBONE_STRIDES)configr)rr(rcompute_backbone_shapesMs   r0Tc Cs|\}}} dt||d} dt||d} tj|d| d|d|} t| dd| |d} td | } tj|||fd | d |d | } t| d d| |d} td | } tj| d| d |d| } t| d d| |d} t| |g} tjd dt||dd| } | S)a9The identity_block is the block that has no conv layer at shortcut # Arguments input_tensor: input tensor kernel_size: default 3, the kernel size of middle conv layer at main path filters: list of integers, the nb_filters of 3 conv layer at main path stage: integer, current stage label, used for generating layer names block: 'a','b'..., current block label, used for generating layer names use_bias: Boolean. To use or not use a bias in conv layers. train_bn: Boolean. Train or freeze Batch Norm layers res_branchbnr$r$2anameuse_biasr7rrelusame2bpaddingr7r82c_outrKLConv2Dr ActivationAdd) input_tensor kernel_sizefiltersstageblockr8train_bn nb_filter1 nb_filter2 nb_filter3conv_name_base bn_name_basexrrridentity_blockks0 "rRrTcCsN|\}} } dt||d} dt||d} tj|d|| d|d|} t| dd| |d} td | } tj| ||fd | d |d | } t| d d| |d} td | } tj| d| d |d| } t| d d| |d} tj| d|| d|d|}t| dd||d}t| |g} tjd dt||dd| } | S)aconv_block is the block that has a conv layer at shortcut # Arguments input_tensor: input tensor kernel_size: default 3, the kernel size of middle conv layer at main path filters: list of integers, the nb_filters of 3 conv layer at main path stage: integer, current stage label, used for generating layer names block: 'a','b'..., current block label, used for generating layer names use_bias: Boolean. To use or not use a bias in conv layers. train_bn: Boolean. Train or freeze Batch Norm layers Note that from stage 3, the first conv layer at main path is with subsample=(2,2) And the shortcut should have subsample=(2,2) as well r1r2r3r4r5stridesr7r8r9rr:r;r<r=r?r61r@rA)rFrGrHrIrJrVr8rKrLrMrNrOrPrQZshortcutrrr conv_blocksP "rXFc Cs|dvs Jtd|}tjdddddd|}td d ||d }td |}tjddd d|}}t|dgdddd|d}t|dgddd|d}t|dgddd|d}}t|dgddd|d}t|dgddd|d}t|dgddd|d}t|dgddd|d}}t|dgddd|d}ddd|}t|D]&} t|dgddt d| |d}qJ|} |rt|dgddd|d}t|dgddd|d}t|dgddd|d} }nd } |||| | gS)!zBuild a ResNet graph. architecture: Can be resnet50 or resnet101 stage5: Boolean. If False, stage5 of the network is not created train_bn: Boolean. Train or freeze Batch Norm layers r"rZ@)r\rSZconv1TrUZbn_conv1r9rr:r;)rVr>rZ)r[r[rTar4)rIrJrVrKb)rIrJrKc)rad)r]r]b)rbrbiN) rBZ ZeroPadding2DrCrrD MaxPooling2DrXrRrangechr) input_imageZ architecturestage5rKrQZC1C2C3Z block_countiC4C5rrr resnet_graphsD     rsc Cs|dddf|dddf}|dddf|dddf}|dddfd|}|dddfd|}||dddf|7}||dddf|7}|t|dddf9}|t|dddf9}|d|}|d|}||}||} tj|||| gddd} | S) zApplies the given deltas to the given boxes. boxes: [N, (y1, x1, y2, x2)] boxes to update deltas: [N, (dy, dx, log(dh), log(dw))] refinements to apply NrTrrZr$?Zapply_box_deltas_outaxisr7)tfZexpstack) boxesdeltasZheightwidthZcenter_yZcenter_xy1x1y2x2resultrrrapply_box_deltas_graphs    rc Cst|d\}}}}tj|ddd\}}}} tt|||}tt|||}tt|||}tt| ||} tj|||| gddd} | | jddf| S)zQ boxes: [N, (y1, x1, y2, x2)] window: [4] in the form y1, x1, y2, x2 rer$rvZ clipped_boxesrur)rwsplitmaximumminimumconcat set_shaper ) rywindowwy1wx1wy2wx2r|r}r~rZclippedrrrclip_boxes_graph srcs2eZdZdZd fdd ZddZddZZS) ProposalLayera6Receives anchor scores and selects a subset to pass as proposals to the second stage. Filtering is done based on anchor scores and non-max suppression to remove overlaps. It also applies bounding box refinement deltas to anchors. Inputs: rpn_probs: [batch, num_anchors, (bg prob, fg prob)] rpn_bbox: [batch, num_anchors, (dy, dx, log(dh), log(dw))] anchors: [batch, num_anchors, (y1, x1, y2, x2)] anchors in normalized coordinates Returns: Proposals in normalized coordinates [batch, rois, (y1, x1, y2, x2)] Nc s,tt|jfi|||_||_||_dSN)rr__init__r/proposal_count nms_threshold)rrrr/kwargsrrrrAszProposalLayer.__init__c s@|ddddddf}|d}|tjjgd}|d}tjjt|d}tjj ||dddj }t ||gdd jj }t ||gd d jj }t j ||gd d jj d gd }t j ||gdd jj dgd }tjgdtjdt j |fdd jj dgd }fdd} t ||g| jj } | S)Nrr$)r$r$rerTTZ top_anchors)sortedr7cSs t||SrrwgatherrQyrrrUz$ProposalLayer.call..cSs t||SrrrrrrrXrcSs t||Srr)r^rQrrrr\rpre_nms_anchorsnamescSs t||Sr)rrrrrrerZrefined_anchors)rrr$r$r cs t|Sr)rrQ)rrrrorZrefined_anchors_clippedcsZtjj||jjdd}t||}tjt|dd}t|d|fdg}|S)NZrpn_non_max_suppressionr9rrr) rwimagenon_max_suppressionrrrrr pad)ryscoresindices proposalsr>rrrnmsys zProposalLayer.call..nms)r-reshaper/RPN_BBOX_STD_DEVrwrZ PRE_NMS_LIMITr nntop_krutils batch_sliceIMAGES_PER_GPUrfloat32) rrrrzanchorsZ pre_nms_limitixrryrrr)rrrrGsD    zProposalLayer.callcCs d|jdfS)Nre)rrZ input_shaperrrcompute_output_shapesz"ProposalLayer.compute_output_shape)Nrrrr rrrr!rrrrr2sCrcCst|tdS)z@Implementation of Log2. TF doesn't have a native implementation.g@)rwrrrrr log2_graphsrcs0eZdZdZfddZddZddZZS)PyramidROIAlignaImplements ROI Pooling on multiple levels of the feature pyramid. Params: - pool_shape: [pool_height, pool_width] of the output pooled regions. Usually [7, 7] Inputs: - boxes: [batch, num_boxes, (y1, x1, y2, x2)] in normalized coordinates. Possibly padded with zeros if not enough boxes to fill the array. - image_meta: [batch, (meta data)] Image details. See compose_image_meta() - feature_maps: List of feature maps from different levels of the pyramid. Each is [batch, height, width, channels] Output: Pooled regions in the shape: [batch, num_boxes, pool_height, pool_width, channels]. The width and height are those specific in the pool_shape in the layer constructor. c s$tt|jfi|t||_dSr)rrrtuple pool_shape)rrrrrrrszPyramidROIAlign.__init__c CsZ|d}|d}|dd}tj|ddd\}}}}||} ||} t|dd} t| d| dtj} tt| | dt| } tdtddtt | tj } t | d} g}g}t t dd D]\}}tt| |}t||}t|dddftj }||t|}t|}|tjj|||||jd d qtj|dd}tj|dd}tt t|dd}tjt|tj |gdd}|dddfd |dddf}tjj|t|dd jddd}t|dddf|}t||}tjt|ddt|ddgdd}t||}|S)Nrr$rTrerr)gl@rfZbilinear)methodi)k)rwrparse_image_meta_graphcastrrZsqrtrrroundint32squeeze enumeraterjwhereequal gather_ndappendZ stop_gradientrcrop_and_resizerr expand_dimsr rrrrr)rrry image_meta feature_mapsr|r}r~rhwr)Z image_areaZ roi_levelZpooledZ box_to_levelrplevelrZ level_boxesZ box_indicesZ box_rangeZsorting_tensorr rrrrsT         $& . zPyramidROIAlign.callcCs$|ddd|j|ddfS)NrrTr)rrrrrrsz$PyramidROIAlign.compute_output_shaperrrrrrs Vrc Cs&ttt|dddt|dgddg}t|t|ddg}tj|ddd\}}}}tj|ddd\}} } } t||} t|| } t|| }t|| }t|| dt|| d}||||}| || | }|||}||}t|t|dt|dg}|S)z`Computes IoU overlaps between two sets of boxes. boxes1, boxes2: [N, (y1, x1, y2, x2)]. r$rrrer)rwrZtilerr rrr)Zboxes1Zboxes2Zb1Zb2Zb1_y1Zb1_x1Zb1_y2Zb1_x2Zb2_y1Zb2_x1Zb2_y2Zb2_x2r|r}r~r intersectionZb1_areaZb2_areaunionZiouoverlapsrrroverlaps_graphs"(      $rc.s6tjtt|dd|gddg}t|t|}Wdn1sP0Yt|dd\}}t|dd\}}tj||dd}tj|t |dddfdd d }t |dkdddf}t |dkdddf} t||} t|| }t|| }tj|| dd }t ||} t || } tj | d d } | d k}tj | d d }|dk}t |dddf}t t |dk|dddf}t |j|j}t|d|}t|d}d|j}t|t|tjtj|}t|d|}t||}t||}t| |tjttd dfddddd}t||}t||}t||}||j}tt|gdd}t||}|}|jr.tj|dd d \}} }!}"tj|dd d \}#}$}%}&|%|#}'|&|$}(||#|'}| |$|(} |!|#|'}!|"|$|(}"t|| |!|"gd }tdt|d})tjt|tj||)|j }*tj!|*dd }*t"|*}*tj||gdd }+t|d},t#|jt|+dd}-t$|+d|-fdg}+t$|d|,|-fdg}t$|d|,|-fg}t$|d|,|-fdg}t$|*d|,|-gddg}*|+|||*fS)aGenerates detection targets for one image. Subsamples proposals and generates target class IDs, bounding box deltas, and masks for each. Inputs: proposals: [POST_NMS_ROIS_TRAINING, (y1, x1, y2, x2)] in normalized coordinates. Might be zero padded if there are not enough proposals. gt_class_ids: [MAX_GT_INSTANCES] int class IDs gt_boxes: [MAX_GT_INSTANCES, (y1, x1, y2, x2)] in normalized coordinates. gt_masks: [height, width, MAX_GT_INSTANCES] of boolean type. Returns: Target ROIs and corresponding class IDs, bounding box shifts, and masks. rois: [TRAIN_ROIS_PER_IMAGE, (y1, x1, y2, x2)] in normalized coordinates class_ids: [TRAIN_ROIS_PER_IMAGE]. Integer class IDs. Zero padded. deltas: [TRAIN_ROIS_PER_IMAGE, (dy, dx, log(dh), log(dw))] masks: [TRAIN_ROIS_PER_IMAGE, height, width]. Masks cropped to bbox boundaries and resized to neural network output size. Note: Returned arrays might be zero padded if not enough target ROIs. rZ roi_assertionr9NZtrim_proposalsZ trim_gt_boxesZtrim_gt_class_idsrTZ trim_gt_masksrurr$MbP?rt?cstjddS)Nr$r)rwargmaxrZpositive_overlapsrrrrz)detection_targets_graph..cSsttgtjSr)rwrconstantint64rrrrrr)Ztrue_fnZfalse_fn)rTrr$rrerZr)%rwZAssertZgreaterr Zcontrol_dependenciesZidentitytrim_zeros_graph boolean_maskrrrZ reduce_maxZ logical_andr%TRAIN_ROIS_PER_IMAGEROI_POSITIVE_RATIOZrandom_shufflerrrZcondrZbox_refinement_graph BBOX_STD_DEVr transpose USE_MINI_MASKrrrjrr MASK_SHAPErrrr).r gt_class_idsgt_boxesgt_masksr/Zasserts_ non_zeroscrowd_ix non_crowd_ix crowd_boxesrcrowd_overlaps crowd_iou_max no_crowd_boolZ roi_iou_maxZpositive_roi_boolZpositive_indicesZnegative_indicesZpositive_countrZnegative_countZ positive_roisZ negative_roisZroi_gt_box_assignment roi_gt_boxesroi_gt_class_idsrzZtransposed_masksZ roi_masksryr|r}r~rgt_y1gt_x1gt_y2gt_x2gt_hgt_wZbox_idsmasksroisNPrrrdetection_targets_graph-s (     "               rcs:eZdZdZfddZddZddZd d d ZZS) DetectionTargetLayera)Subsamples proposals and generates target box refinement, class_ids, and masks for each. Inputs: proposals: [batch, N, (y1, x1, y2, x2)] in normalized coordinates. Might be zero padded if there are not enough proposals. gt_class_ids: [batch, MAX_GT_INSTANCES] Integer class IDs. gt_boxes: [batch, MAX_GT_INSTANCES, (y1, x1, y2, x2)] in normalized coordinates. gt_masks: [batch, height, width, MAX_GT_INSTANCES] of boolean type Returns: Target ROIs and corresponding class IDs, bounding box shifts, and masks. rois: [batch, TRAIN_ROIS_PER_IMAGE, (y1, x1, y2, x2)] in normalized coordinates target_class_ids: [batch, TRAIN_ROIS_PER_IMAGE]. Integer class IDs. target_deltas: [batch, TRAIN_ROIS_PER_IMAGE, (dy, dx, log(dh), log(dw)] target_mask: [batch, TRAIN_ROIS_PER_IMAGE, height, width] Masks cropped to bbox boundaries and resized to neural network output size. Note: Returned arrays might be zero padded if not enough target ROIs. c s tt|jfi|||_dSr)rrrr/rr/rrrrrszDetectionTargetLayer.__init__csR|d}|d}|d}|d}gd}tj||||gfddjj|d}|S) Nrr$rTrZ)rtarget_class_ids target_bbox target_maskcst||||jSr)rr/)rrQrzrrrrrz+DetectionTargetLayer.call..r)rrr/r)rrrrrrroutputsrrrrs  zDetectionTargetLayer.callcCsDd|jjdfd|jjfd|jjdfd|jj|jjd|jjdfgS)Nrerr$)r/rrrrrrrs     z)DetectionTargetLayer.compute_output_shapeNcCsgdS)N)NNNNr)rrmaskrrr compute_masksz!DetectionTargetLayer.compute_mask)N) rrrr rrrrr!rrrrrs   rcs<tj|dtjd}tjt|jd|gdd}t||}t||}t||j} t | |} t |dkdddfj rt |j kdddf} tj tdt| dtdt|t|t| td} fdd} tj| | tjd} t| d g} t| t | d kdddf} tj tdt| dtdj}t|}tt|d|}tjj||d d d}t|tjt| tt|d tjft|d tjfgdd}jt|d}t|d|fd gd}|S)aSRefine classified proposals and filter overlaps and return final detections. Inputs: rois: [N, (y1, x1, y2, x2)] in normalized coordinates probs: [N, num_classes]. Class probabilities. deltas: [N, num_classes, (dy, dx, log(dh), log(dw))]. Class-specific bounding box deltas. window: (y1, x1, y2, x2) in normalized coordinates. The part of the image that contains the image excluding the padding. Returns detections shaped: [num_detections, (y1, x1, y2, x2, class_id, score)] where coordinates are normalized. r$)rvZ output_typerrNcstt|dddf}tjjt|t|jjd}tt||}jt|d}tj |d|fgddd}| jg|S)z9Apply Non-Maximum Suppression on ROIs of the given class.Nr)Zmax_output_sizeZ iou_thresholdCONSTANTr)modeZconstant_values) rwrrrrrDETECTION_MAX_INSTANCESZDETECTION_NMS_THRESHOLDr rr)class_idZixsZ class_keepgapr/keepZpre_nms_class_idsZ pre_nms_roisZpre_nms_scoresrr nms_keep_map*s  z-refine_detections_graph..nms_keep_maprrT)rr.rr)rwrrrxrjr rrrrrZDETECTION_MIN_CONFIDENCEZsetsZset_intersectionrZsparse_tensor_to_denseruniqueZmap_fnrrrrrrrZto_floatnewaxisr)rZprobsrzrr/ class_idsrZ class_scoresZdeltas_specificZ refined_roisZ conf_keepZunique_pre_nms_class_idsr Znms_keepZ roi_countZclass_scores_keepZnum_keepZtop_ids detectionsrrrrrefine_detections_graphsP      "    rcs2eZdZdZd fdd ZddZddZZS) DetectionLayerzTakes classified proposal boxes and their bounding box deltas and returns the final detection boxes. Returns: [batch, num_detections, (y1, x1, y2, x2, class_id, class_score)] where coordinates are normalized. Nc s tt|jfi|||_dSr)rrrr/rrrrriszDetectionLayer.__init__c s|d}|d}|d}|d}t|}|dd}t|d|dd}t||||gfddjj} t| jjjj d gS) Nrr$rTrZr)rcst||||jSr)rr/)rQrrrrrrr~rz%DetectionLayer.call..r) rnorm_boxes_graphrrr/rrwr BATCH_SIZEr) rrr mrcnn_class mrcnn_bboxrmr)rZdetections_batchrrrrms     zDetectionLayer.callcCsd|jjdfS)Nr)r/rrrrrrsz#DetectionLayer.compute_output_shape)Nrrrrrr`srcCstjdddd|dd|}tjd|dd d d d |}td d|}tjddd|}tj|ddd d dd |}tdd|}|||gS)aBuilds the computation graph of Region Proposal Network. feature_map: backbone features [batch, height, width, depth] anchors_per_location: number of anchors per pixel in the feature map anchor_stride: Controls the density of anchors. Typically 1 (anchors for every pixel in the feature map), or 2 (every other pixel). Returns: rpn_class_logits: [batch, H * W * anchors_per_location, 2] Anchor classifier logits (before softmax) rpn_probs: [batch, H * W * anchors_per_location, 2] Anchor classifier probabilities. rpn_bbox: [batch, H * W * anchors_per_location, (dy, dx, log(dh), log(dw))] Deltas to be applied to anchors. rbrYr;r:Zrpn_conv_shared)r> activationrVr7rTr4validlinearZ rpn_class_raw)r>rr7cSst|t|dddgS)NrrrTrwrr trrrrrzrpn_graph..softmaxZ rpn_class_xxxr9reZ rpn_bbox_predcSst|t|dddgS)Nrrrerrrrrrr)rBrCLambdarD)Z feature_mapanchors_per_location anchor_stridesharedrQrpn_class_logitsZ rpn_probsrpn_bboxrrr rpn_graphs@   r#cCs2tjdd|gdd}t|||}tj|g|ddS)a,Builds a Keras model of the Region Proposal Network. It wraps the RPN graph so it can be used multiple times with shared weights. anchors_per_location: number of anchors per pixel in the feature map anchor_stride: Controls the density of anchors. Typically 1 (anchors for every pixel in the feature map), or 2 (every other pixel). depth: Depth of the backbone feature map. Returns a Keras Model object. The model outputs, when called, are: rpn_class_logits: [batch, H * W * anchors_per_location, 2] Anchor classifier logits (before softmax) rpn_probs: [batch, H * W * anchors_per_location, 2] Anchor classifier probabilities. rpn_bbox: [batch, H * W * anchors_per_location, (dy, dx, log(dh), log(dw))] Deltas to be applied to anchors. NZinput_rpn_feature_mapr r7Z rpn_modelr9)rBInputr#KMModel)rrZdepthZinput_feature_maprrrrbuild_rpn_models   r(rdc Cs:t||gdd||g|}tjtj|||fdddd|}tjtdd||d}td|}tjt|d d d|}tjtd d||d}td|}tjd d dd|}tjt|dd|} tjtddd| } tjtj|ddddd|}t |} tj | d|dfdd|} | | | fS)aBuilds the computation graph of the feature pyramid network classifier and regressor heads. rois: [batch, num_rois, (y1, x1, y2, x2)] Proposal boxes in normalized coordinates. feature_maps: List of feature maps from different layers of the pyramid, [P2, P3, P4, P5]. Each has a different resolution. image_meta: [batch, (meta data)] Image details. See compose_image_meta() pool_size: The width of the square feature map generated from ROI Pooling. num_classes: number of classes, which determines the depth of the results train_bn: Boolean. Train or freeze Batch Norm layers fc_layers_size: Size of the 2 FC layers Returns: logits: [batch, num_rois, NUM_CLASSES] classifier logits (before softmax) probs: [batch, num_rois, NUM_CLASSES] classifier probabilities bbox_deltas: [batch, num_rois, NUM_CLASSES, (dy, dx, log(dh), log(dw))] Deltas to apply to proposal boxes Zroi_align_classifierr9rr>Zmrcnn_class_conv1Zmrcnn_class_bn1rr:r4Zmrcnn_class_conv2Zmrcnn_class_bn2cSstt|ddS)NrZrT)Krrrrrrrz&fpn_classifier_graph..Z pool_squeezemrcnn_class_logitsrrrer)rZ mrcnn_bbox_fcr$r) rrBTimeDistributedrCrrDrZDenser* int_shapeZReshape) rrr pool_size num_classesrKfc_layers_sizerQr r+Z mrcnn_probssrrrrfpn_classifier_graphsB   r2cCspt||gdd||g|}tjtjdddddd|}tjtdd||d }td |}tjtjddddd d|}tjtd d||d }td |}tjtjddddd d|}tjtdd||d }td |}tjtjdddddd|}tjtdd||d }td |}tjtjdddd ddd|}tjtj|dddddd|}|S)aBuilds the computation graph of the mask head of Feature Pyramid Network. rois: [batch, num_rois, (y1, x1, y2, x2)] Proposal boxes in normalized coordinates. feature_maps: List of feature maps from different layers of the pyramid, [P2, P3, P4, P5]. Each has a different resolution. image_meta: [batch, (meta data)] Image details. See compose_image_meta() pool_size: The width of the square feature map generated from ROI Pooling. num_classes: number of classes, which determines the depth of the results train_bn: Boolean. Train or freeze Batch Norm layers Returns: Masks [batch, num_rois, MASK_POOL_SIZE, MASK_POOL_SIZE, NUM_CLASSES] Zroi_align_maskr9r]rYr;r)Zmrcnn_mask_conv1Zmrcnn_mask_bn1rr:Zmrcnn_mask_conv2Zmrcnn_mask_bn2Zmrcnn_mask_conv3Zmrcnn_mask_bn3Zmrcnn_mask_conv4Zmrcnn_mask_bn4rSrT)rVrZmrcnn_mask_deconvr4r$Zsigmoid mrcnn_mask)rrBr,rCrrDZConv2DTranspose)rrrr.r/rKrQrrrbuild_fpn_mask_graph,sX r4cCsFt||}tt|dd}|d|dd||d}|S)zdImplements Smooth-L1 loss. y_true and y_pred are typically: [N, 4], but could be any shape. rrrtrTr$)r*absrZless)y_truey_predZdiffZ less_than_onelossrrrsmooth_l1_lossks r9cCst|d}tt|dtj}tt|d}t||}t||}tj ||dd}t t |dkt |t d}|S)zRPN anchor classifier loss. rpn_match: [batch, anchors, 1]. Anchor match type. 1=positive, -1=negative, 0=neutral anchor. rpn_class_logits: [batch, anchors, 2]. RPN classifier logits for BG/FG. rr$rT)targetoutputZ from_logits)rwrr*rrrrZ not_equalrZsparse_categorical_crossentropyswitchr meanr) rpn_matchr!Z anchor_classrr8rrrrpn_class_loss_graphus   $r@cCst|d}tt|d}t||}tjtt|dtjdd}t |||j }t ||}t t |dkt|td}|S)aReturn the RPN bounding box loss graph. config: the model config object. target_bbox: [batch, max positive anchors, (dy, dx, log(dh), log(dw))]. Uses 0 padding to fill in unsed bbox deltas. rpn_match: [batch, anchors, 1]. Anchor match type. 1=positive, -1=negative, 0=neutral anchor. rpn_bbox: [batch, anchors, (dy, dx, log(dh), log(dw))] rr$rrr<)r*rrwrrrsumrrbatch_pack_graphrr9r=r r>r)r/rr?r"rZ batch_countsr8rrrrpn_bbox_loss_graphs    $rCcCsZt|d}tj|dd}t|d|}tjj||d}||}t|t|}|S)aLoss for the classifier head of Mask RCNN. target_class_ids: [batch, num_rois]. Integer class IDs. Uses zero padding to fill in the array. pred_class_logits: [batch, num_rois, num_classes] active_class_ids: [batch, num_classes]. Has a value of 1 for classes that are in the dataset of the image, and 0 for classes that are not in the dataset. rrTrr)labelsZlogits)rwrrrrZ(sparse_softmax_cross_entropy_with_logits reduce_sum)rZpred_class_logitsactive_class_idsZpred_class_idsZ pred_activer8rrrmrcnn_class_loss_graphs rGcCst|d}t|d}t|dt|ddf}t|dkdddf}tt||tj}tj||gdd }t||}t ||}t t |dkt ||d t d }t|}|S) zLoss for Mask R-CNN bounding box refinement. target_bbox: [batch, num_rois, (dy, dx, log(dh), log(dw))] target_class_ids: [batch, num_rois]. Integer class IDs. pred_bbox: [batch, num_rois, num_classes, (dy, dx, log(dh), log(dw))] r)rrerrTrerNr$r)r6r7r<)r*rr-rwrrrrrxrr=r r9rr>)rrZ pred_bboxZpositive_roi_ixZpositive_roi_class_idsrr8rrrmrcnn_bbox_loss_graphs"       rIc Cst|d}t|}t|d|d|df}t|}t|d|d|d|df}t|gd}t|dkdddf}tt||tj}tj ||gd d }t||}t ||} t t |dktj || d td } t| } | S) axMask binary cross-entropy loss for the masks head. target_masks: [batch, num_rois, height, width]. A float32 tensor of values 0 or 1. Uses zero padding to fill array. target_class_ids: [batch, num_rois]. Integer class IDs. Zero padded. pred_masks: [batch, proposals, height, width, num_classes] float32 tensor with values from 0 to 1. rHrrTrZre)rrZr$rTrNr$r)r:r;r<)r*rrwr rrrrrrxrr=r Zbinary_crossentropyrr>) Z target_masksrZ pred_masks mask_shapeZ pred_shapeZ positive_ixZpositive_class_idsrr6r7r8rrrmrcnn_mask_loss_graphs(        rKcs||}||\}}} | r"dS|j} tj||j|j|j|jd\}} } } }t || | |}|rt dt ddrt|}t|}|rddl}gdfdd }|j}|j}|}||}|j|tj|j|d d }|j|ksJd |j|ksJd |tj}tj|dddk}|dddd|f}||}t|}tj|jgtjd}|j|j|d}d||<|rt|||j}t || |j| | |}|||||dfS)aLoad and return ground truth data for an image (image, mask, bounding boxes). augment: (deprecated. Use augmentation instead). If true, apply random image augmentation. Currently, only horizontal flipping is offered. augmentation: Optional. An imgaug (https://github.com/aleju/imgaug) augmentation. For example, passing imgaug.augmenters.Fliplr(0.5) flips images right/left 50% of the time. use_mini_mask: If False, returns full-size masks that are the same height and width as the original image. These can be big, for example 1024x1024x100 (for 100 instances). Mini masks are smaller, typically, 224x224 and are generated by extracting the bounding box of the object and resizing it to MINI_MASK_SHAPE. Returns: image: [height, width, 3] shape: the original shape of the image before resizing and cropping. class_ids: [instance_count] Integer class IDs bbox: [instance_count, (y1, x1, y2, x2)] mask: [height, width, instance_count]. The height and width are those of the image unless use_mini_mask is True, in which case they are defined in MINI_MASK_SHAPE. )NNNNNr$Zmin_dimZ min_scaleZmax_dimrz4'augment' is deprecated. Use 'augmentation' instead.rr$N) Z SequentialZSomeOfZOneOfZ SometimesZFliplrZFlipudZ CropAndPadZAffineZPiecewiseAffinecs |jjvS)z.Determines which augmenters to apply to masks.)rr)imagesZ augmenterparentsdefaultZMASK_AUGMENTERSrrhookdszload_image_gt..hook)Z activator)Zhooksz(Augmentation shouldn't change image sizez'Augmentation shouldn't change mask sizer#rrsource)!Z load_imageZget_maskr r resize_image IMAGE_MIN_DIMIMAGE_MIN_SCALE IMAGE_MAX_DIMIMAGE_RESIZE_MODEZ resize_maskloggingZwarningrandomrandintr-ZfliplrimgaugZto_deterministicZ augment_imageastypeuint8Z HooksImagesboolrAZextract_bboxeszerosr/rsource_class_ids image_infoZ minimize_maskMINI_MASK_SHAPEcompose_image_meta)datasetr/image_idaugment augmentation use_mini_maskrrr errorZoriginal_shaperscaler>cropr[rQr)rJZdetZ_idxZbboxrFr`rrrPr load_image_gtsZ         rlc.Cs|jddksJ|jtjks.Jd|j|jtjksJJd|j|jtjksfJd|jt|dkd}|jddksJd||}||}|dddd|f}|dddf|dddf|dddf|dddf}|dddf|dddf|dddf|dddf}t|jd|jdf}t|jdD].} || } t | ||| ||dd| f<q\tj |dd } |t |jd| f} || } || }t| d kd}t| d kd}t |j|j}|jd|krtjj||d d }n|}|j|jd}|jd|krRtjj||d d }n|}t||g}|j|jd}|dkr |jddkrt| d kd}|jd|ksJtjj||d d }|jd|ksJt||g}n tjj||d d }t||g}|jd|jks4Jd|jd|jd| |ddf<d||<||}| |}||}| |}tj|j|jdftjd}t|dkd}t ||||ddf||||f<||j}tj|j|jd|jd|jftjd}|D]} || }|dksJd|| }|dddd|f} |jrtj|jddtd}!||\}"}#}$}%|%|#}&|$|"}'tt | |'|&ft|!|"|$|#|%f<|!} || tj\}(})}*}+| |(|*|)|+f},t |,|j}-|-|| dddd|f<q||||fS)auGenerate targets for training Stage 2 classifier and mask heads. This is not used in normal training. It's useful for debugging or to train the Mask RCNN heads without using the RPN head. Inputs: rpn_rois: [N, (y1, x1, y2, x2)] proposal boxes. gt_class_ids: [instance count] Integer class IDs gt_boxes: [instance count, (y1, x1, y2, x2)] gt_masks: [height, width, instance count] Ground truth masks. Can be full size or mini-masks. Returns: rois: [TRAIN_ROIS_PER_IMAGE, (y1, x1, y2, x2)] class_ids: [TRAIN_ROIS_PER_IMAGE]. Integer class IDs. bboxes: [TRAIN_ROIS_PER_IMAGE, NUM_CLASSES, (y, x, log(h), log(w))]. Class-specific bbox refinements. masks: [TRAIN_ROIS_PER_IMAGE, height, width, NUM_CLASSES). Class specific masks cropped to bbox boundaries and resized to neural network output size. rzExpected int but got {}zExpected bool but got {}zImage must contain instances.NrTrZr$rrtFreplaceTz(keep doesn't match ROI batch size {}, {}rerzclass id must be greater than 0)r r r-rrZbool_rr_rjrZ compute_iouraranger%rrrYchoiceZ concatenate NUM_CLASSESrZbox_refinementrrr IMAGE_SHAPEr^rZresizer\).rpn_roisrrrr/Z instance_idsZ rpn_roi_areaZ gt_box_arearrpgtZrpn_roi_iou_argmaxZrpn_roi_iou_maxZrpn_roi_gt_boxesZrpn_roi_gt_class_idsZfg_idsZbg_idsZ fg_roi_countZ keep_fg_idsZ remainingZ keep_bg_idsr Zkeep_extra_idsrrrZroi_gt_assignmentZbboxesZpos_idsrrZgt_idZ class_maskZ placeholderrrrrrrr|r}r~rrrrrrbuild_detection_targetss@@$    rucCstj|jdgtjd}t|jdf}t|dkd}|jddkrt|dkd}||} ||}||}t|| } tj| dd} | dk} ntj |jdgt d} t||} tj | dd}| t | jd|f}d||dk| @<t | tj| ddkd d df}d||<d||d k<t|dkd}t||jd }|dkrntjj||d d }d||<t|dkd}t||jt|dk}|dkrtjj||d d }d||<t|dkd}d}t|||D]\}}|||}|d |d}|d|d}|dd|}|dd|}|d |d}|d|d}|dd|}|dd|}||||||t||t||g||<|||j<|d7}q||fS)aGiven the anchors and GT boxes, compute overlaps and identify positive anchors and deltas to refine them to match their corresponding GT boxes. anchors: [num_anchors, (y1, x1, y2, x2)] gt_class_ids: [num_gt_boxes] Integer class IDs. gt_boxes: [num_gt_boxes, (y1, x1, y2, x2)] Returns: rpn_match: [N] (int32) matches between anchors and GT boxes. 1 = positive anchor, -1 = negative anchor, 0 = neutral rpn_bbox: [N, (dy, dx, log(dh), log(dw))] Anchor bbox deltas. rrrer$rrrg333333?Ngffffff?rTFrmrZrt)r-r_r rRPN_TRAIN_ANCHORS_PER_IMAGErrZcompute_overlapsZamaxZonesr^rroZargwherer lenrYrprAziprr)r)rrrr/r?r"rrrrrrrZanchor_iou_argmaxZanchor_iou_maxZ gt_iou_argmaxidsZextrarrpr^rtrrZ gt_center_yZ gt_center_xZa_hZa_wZ a_center_yZ a_center_xrrrbuild_rpn_targets7sb   $         rzcCs tj|dftjd}td||jd}t|jdD]}||\}}} } | |} | |} t|| d} t| | |d}t|| d}t| | |d}tj | ||ddf}tj |||ddf}d}|t |dddf|dddf|kd|}|t |dddf|dddf|kd|}|jd|kr|jd|krq`qtj tj |ddddd\}}tj tj |ddddd\}}t ||||g}||||||d<q8|||jd}tj d|d|ddf}tj d|d|ddf}d}|t |dddf|dddf|kd|}|t |dddf|dddf|kd|}|jd|kr|jd|krڐqqtj tj |ddddd\}}tj tj |ddddd\}}t ||||g}||| d<|S) a]Generates ROI proposals similar to what a region proposal network would generate. image_shape: [Height, Width, Depth] count: Number of ROIs to generate gt_class_ids: [N] Integer ground truth class IDs gt_boxes: [N, (y1, x1, y2, x2)] Ground truth boxes in pixels. Returns: [count, (y1, x1, y2, x2)] ROI boxes in pixels. rerg?rr$rTNr)r-r_rr%r rjr r rYrZr5rsortZhstack)r)countrrrZ rois_per_boxrprrrrrrZr_y1Zr_y2Zr_x1Zr_x2Zy1y2Zx1x2Z thresholdr}rr|r~Zbox_roisZremaining_countZ global_roisrrrgenerate_random_roissD 6666 r}r$c - csd} d} t|j} d} |pg}t||j} t|j|j| |j |j }z| dt | } |rr| dkrrtj | | | }|j|d|vrt||||d|jd\}}}}}}n"t||||||jd\}}}}}}|rddl}|jd|dd WqFt|dks WqFt|j||||\}}|rTt|j|||}|rTt|||||\}}}}| dkrtj|f|j|jd }tj||jddg|jd }tj||jd g|jd } tj|f|jtjd }!tj||jftjd }"tj||jd ftjd }#tj||jd|jd|jf|jd }$|rtj||jdd f|jd }%|rtj|f|j|jd }&tj|f|j|jd }'tj|f|j|jd }(tj|f|j|jd })|jd|jkrtj jt |jd|jd d }*||*}||*}|dddd|*f}||| <|ddtj!f|| <|| | <t"|#tj||!| <||"| d|jdf<||#| d|jdf<||$| ddddd|jdf<|r||%| <|r||&| <||'| <||(| <||)| <| d7} | |kr*|!||| |"|#|$g}+g},|r|+$|%g|r|+$|&gt%|'d}'|,$|'|(|)g|+|,fVd} WqFt&t'fyFYqFt()d|j|| d7} | dkrvYqF0qFdS)aA generator that returns images and corresponding target class ids, bounding box deltas, and masks. dataset: The Dataset object to pick data from config: The model config object shuffle: If True, shuffles the samples before every epoch augment: (deprecated. Use augmentation instead). If true, apply random image augmentation. Currently, only horizontal flipping is offered. augmentation: Optional. An imgaug (https://github.com/aleju/imgaug) augmentation. For example, passing imgaug.augmenters.Fliplr(0.5) flips images right/left 50% of the time. random_rois: If > 0 then generate proposals to be used to train the network classifier and mask heads. Useful if training the Mask RCNN part without the RPN. batch_size: How many images to return in each call detection_targets: If True, generate detection targets (class IDs, bbox deltas, and masks). Typically for debugging or visualizations because in trainig detection targets are generated by DetectionTargetLayer. no_augmentation_sources: Optional. List of sources to exclude for augmentation. A source is string that identifies a dataset and is defined in the Dataset class. Returns a Python generator. Upon calling next() on it, the generator returns two lists, inputs and outputs. The contents of the lists differs depending on the received arguments: inputs list: - images: [batch, H, W, C] - image_meta: [batch, (meta data)] Image details. See compose_image_meta() - rpn_match: [batch, N] Integer (1=positive anchor, -1=negative, 0=neutral) - rpn_bbox: [batch, N, (dy, dx, log(dh), log(dw))] Anchor bbox deltas. - gt_class_ids: [batch, MAX_GT_INSTANCES] Integer class IDs - gt_boxes: [batch, MAX_GT_INSTANCES, (y1, x1, y2, x2)] - gt_masks: [batch, height, width, MAX_GT_INSTANCES]. The height and width are those of the image unless use_mini_mask is True, in which case they are defined in MINI_MASK_SHAPE. outputs list: Usually empty in regular training. But if detection_targets is True then the outputs list contains target class_ids, bbox deltas, and masks. rrr$rRN)rfrgrhziProblems while processing masks of image {}. This image is skipped in the training and should be checked.rT) stacklevelrreFrmzError processing image {}rf)*r-copy image_idsr0rrrgenerate_pyramid_anchorsRPN_ANCHOR_SCALESRPN_ANCHOR_RATIOSr.RPN_ANCHOR_STRIDErwrYshufflerarlrwarningswarnranyrzr r}rur_r rvrZMAX_GT_INSTANCESrrpror  mold_imager\extendr GeneratorExitKeyboardInterruptrXZ exception)-rdr/rrfrgZ random_rois batch_sizeZdetection_targetsno_augmentation_sourcesr_Z image_indexrZ error_countbackbone_shapesrrerrrrrrirr?r"rsrZmrcnn_class_idsrr3Zbatch_image_metaZbatch_rpn_matchZbatch_rpn_bboxZ batch_imagesZbatch_gt_class_idsZbatch_gt_boxesZbatch_gt_masksZbatch_rpn_roisZ batch_roisZbatch_mrcnn_class_idsZbatch_mrcnn_bboxZbatch_mrcnn_maskryrrrrrdata_generators\3              "      rc@seZdZdZddZddZddZd*d d Zd d ZddZ d+ddZ d,ddZ d-ddZ ddZ ddZd.ddZd/ddZd d!Zd0d"d#Zd$d%Zd&d'Zd1d(d)Zd S)2MaskRCNNzpEncapsulates the Mask RCNN model functionality. The actual Keras model is in the keras_model property. cCs:|dvs J||_||_||_||j||d|_dS)z mode: Either "training" or "inference" config: A Sub-class of the Config class model_dir: Directory to save training logs and trained weights r inference)rr/N)rr/ model_dir set_log_dirbuild keras_model)rrr/rrrrrs  zMaskRCNN.__init__c9s.|dvs Jjdd\}}|dt|dksF|dt|dkrNtdtjddjdgddtjjgdd}|d kr@tjdd gd tjd }tjdd gdtjd }tjdgdtjd }tjdd gdtjd } t fdd| } j rtjj dj d dgdt d } n"tjjdjd dgdt d } n|dkr\tjdd gdd} t jrjdjd\} }}}}ntjdjd\} }}}}tjjddd|}tjddtjddd|tjjdd d|g}tjd!dtjdd"d|tjjdd#d|g}tjd$dtjdd%d|tjjdd&d|g}tjjd'd(d)d*|}tjjd'd(d+d*|}tjjd'd(d,d*|}tjjd'd(d-d*|}tjddd.d/|}|||||g}||||g}|d kr,|jtjfjtj fd0dd1dn| tjtjj}g}|D]}|||gqNgd2}t t!|}d3d4t!||D}|\}}} |d krj"nj#}!t$|!j%d5d6|| g}"|d krzt d7d|}#j&stjj"d gd8tjd }$t fd9d|$}%n|"}%t'd:d|%|| | g\}&}'}(})t(|&||j)j*jj+d;\}*}+},t,|&||j-j*jd<}-tj d=dd>d|&}.tj d?dd@d||g}/tj fdAddBd||| g}0tj dCddDd|'|*|#g}1tj dEddFd|(|'|,g}2tj dGddHd|)|'|-g}3||||| | g}4j&sH|4|$||| |*|+|,|-|"|.|/|0|1|2|3g}t.j/|4|dId}5nt(|"||j)j*jj+d;\}*}+},t0dJd|"|+|,|g}6t dKd|6}7t,|7||j-j*jd<}-t.j/|| g|6|+|,|-|"|| gdId}5j1d kr*ddLl2m3}8|8|5j1}5|5S)MzBuild Mask R-CNN architecture. input_shape: The shape of the input image. mode: Either "training" or "inference". The inputs and outputs of the model differ accordingly. rNrTr[zImage size must be dividable by 2 at least 6 times to avoid fractions when downscaling and upscaling.For example, use 256, 320, 384, 448, 512, ... etc. rlr$input_image_metarr$input_rpn_match)r r7r reinput_rpn_bboxinput_gt_class_idsinput_gt_boxescst|tddSNr$rZrr*r rrlrrr9rz MaskRCNN.build..rinput_gt_masksr input_anchorsT)rmrKr4Zfpn_c5p5r9Z fpn_p4addrSZfpn_p5upsampled)r r7Zfpn_c4p4Z fpn_p3addZfpn_p4upsampledZfpn_c3p3Z fpn_p2addZfpn_p3upsampledZfpn_c2p2rYZSAMEZfpn_p2)r>r7Zfpn_p3Zfpn_p4Zfpn_p5Zfpn_p6)r.rVr7cs tSr)rwZVariabler)rrrrrr)r! rpn_classr"cSs&g|]\}}tjd|dt|qS)r$ru)rBZ Concatenatelist)r'onrrrr*sz"MaskRCNN.build..ZROI)rrr7r/cSs t|dS)NrF)rrrrrrrZ input_roicst|tddSrrrrrrrrZproposal_targets)rKr0)rKcSs|dS)Nr$rrrrrrr output_roiscSst|Sr)r@rrrrrrrpn_class_losscstg|RSr)rCr)r/rrrr rpn_bbox_losscSst|Sr)rGrrrrrrmrcnn_class_losscSst|Sr)rIrrrrrrmrcnn_bbox_losscSst|Sr)rKrrrrrrmrcnn_mask_loss mask_rcnnZmrcnn_detectioncSs|dddfS)N.rerrrrrr. r) ParallelModel)4rrr% ExceptionrBr%ZIMAGE_META_SIZErwrrrrrbr^r+r,ZTRAIN_BNrsrCZTOP_DOWN_PYRAMID_SIZErEZ UpSampling2Dri get_anchorsr- broadcast_torr r(rrwrrrrxZPOST_NMS_ROIS_TRAININGZPOST_NMS_ROIS_INFERENCErZRPN_NMS_THRESHOLDZ USE_RPN_ROISrr2Z POOL_SIZErqZFPN_CLASSIF_FC_LAYERS_SIZEr4ZMASK_POOL_SIZEr&r'rZ GPU_COUNTZparallel_modelr)9rrr/rrrrrrrrrrrrnrorqrrZP5ZP4ZP3ZP2ZP6Zrpn_feature_mapsZmrcnn_feature_mapsZrpnZ layer_outputspZ output_namesrr!rr"rrsrFZ input_roisZ target_roisrrrrr+rrr3rrrZ class_lossZ bbox_lossZ mask_lossrmodelrZdetection_boxesrr)rr/rlrr s (                                     zMaskRCNN.buildcstt|jd}|jjtfdd|}t|}|sZddl }t |j d |jtj |j|d}tt|d}td d|}t|}|sddl }t |j d |tj ||d}|S) zFinds the last checkpoint file of the last trained model in the model directory. Returns: The path of the last checkpoint file r$cs |Sr startswithfkeyrrrW rz$MaskRCNN.find_last..rNz'Could not find model directory under {}rrTcSs |dS)Nrrrrrrrd rz!Could not find weight files in {})nextoswalkrr/NAMElowerfilterrerrnoFileNotFoundErrorZENOENTrpathjoin)rZ dir_namesrZdir_nameZ checkpointsZ checkpointrrr find_lastN s*   zMaskRCNN.find_lastFNc sddl}zddlm}Wnty6ddlm}Yn0r@d}|durPtd|j|dd}d |jvrxd |vrx|d }|j}t|d r|j j n|j }rt fd d |}|r| ||n | ||t|dr|||dS)zModified version of the corresponding Keras function with the addition of multi-GPU support and the ability to exclude some layers from loading. exclude: list of layer names to exclude rN)saving)topologyTz`load_weights` requires h5py.r)rZ layer_namesZ model_weights inner_modelcs |jvSrr9)lexcluderrr rz'MaskRCNN.load_weights..close)h5py keras.enginer ImportErrorrZFileattrsrhasattrrlayersrZ$load_weights_from_hdf5_group_by_nameZload_weights_from_hdf5_grouprr) rfilepathZby_namerrrrrrrrr load_weightso s2    zMaskRCNN.load_weightscCs$ddlm}d}|d|ddd}|S)z]Downloads ImageNet trained weights from Keras. Returns path to weights file. r)get_filez|https://github.com/fchollet/deep-learning-models/releases/download/v0.2/resnet50_weights_tf_dim_ordering_tf_kernels_notop.h5z4resnet50_weights_tf_dim_ordering_tf_kernels_notop.h5modelsZ a268eb855778b3df3c7506639542a6af)Z cache_subdirZmd5_hash)Zkeras.utils.data_utilsr)rrZTF_WEIGHTS_PATH_NO_TOPZ weights_pathrrrget_imagenet_weights s zMaskRCNN.get_imagenet_weightsc s*tjj||jjd}gj_ij_gd}|D]L}j|}|j jj vrRq2t j |j ddjj |d}j|q2fddjjD}jt |jj|dgtjjd |D]Z}|jjvrqʈj|}jj|t j |j ddjj |d}jj|qdS) zGets the model ready for training. Adds losses, regularization, and metrics. Then calls the Keras compile() function. )ZlrmomentumZclipnorm)rrrrrT)ZkeepdimsrcsHg|]@}d|jvrd|jvrtjjj|tt|tj qS)ZgammaZbeta) r7kerasZ regularizersl2r/Z WEIGHT_DECAYrwrr r)r'rrrrr* s z$MaskRCNN.compile..N) optimizerr8)rZ optimizersZSGDr/ZGRADIENT_CLIP_NORMrZ_lossesZ_per_input_lossesZ get_layerr;ZlossesrwZ reduce_meanZ LOSS_WEIGHTSgetZadd_lossZtrainable_weightsZadd_ncompilerwrZ metrics_namesrZmetrics_tensors) r learning_raterrZ loss_namesr7layerr8Z reg_lossesrrrr sD     zMaskRCNN.compilerr$cCs|dkr|durtd|p |j}t|dr4|jjn|j}|D]}|jjdkrptd|j|j |||ddq>|j sxq>t t ||j}|jjd kr||j_n||_|r>|dkr>td d ||j|jjq>dS) zbSets model layers as trainable if their names match the given regular expression. rNzSelecting layers to trainrr'z In model: re)rindentr,z{}{:20} ({}) )rrrrrrrrr7 set_trainableZweightsr^re fullmatchr trainabler)r layer_regexrrverboserrrrrrr s0       zMaskRCNN.set_trainablec Csd|_tj}|rd}t||}|rtt|dt|dt|dt|dt|d}t|ddd|_td |jtj |j d |j j||_tj |jd |j j|_|jd d |_dS)a.Sets the model log directory and epoch counter. model_path: If None, or a format different from what this code uses then set a new log directory and start epochs from 0. Otherwise, extract the log directory and the epoch counter from the file name. rzS.*[/\\][\w-]+(\d{4})(\d{2})(\d{2})T(\d{2})(\d{2})[/\\]mask\_rcnn\_[\w-]+(\d{4})\.h5r$rTrZrerfrzRe-starting from epoch %dz{}{:%Y%m%dT%H%M}zmask_rcnn_{}_*epoch*.h5z*epoch*z {epoch:04d}N)epochdatetimenowrmatchr%grouprrrrrrr/rrlog_dircheckpoint_pathrn)rZ model_pathrZregexrrrrr s,         zMaskRCNN.set_log_dirc  CsH|jdksJddddddd} || vr6| |}t||jd ||jj|d } t||jd |jjd } tj|jst |jt j j |jd d d dt j j |jd d dg} |r| |7} td|j|td|j|||||jjtjdkrd } nt} |jj| |j||jj| | |jjd| d d t|j||_dS)a6Train the model. train_dataset, val_dataset: Training and validation Dataset objects. learning_rate: The learning rate to train with epochs: Number of training epochs. Note that previous training epochs are considered to be done alreay, so this actually determines the epochs to train in total rather than in this particaular call. layers: Allows selecting wich layers to train. It can be: - A regular expression to match layer names to train - One of these predefined values: heads: The RPN, classifier and mask heads of the network all: All the layers 3+: Train Resnet stage 3 and up 4+: Train Resnet stage 4 and up 5+: Train Resnet stage 5 and up augmentation: Optional. An imgaug (https://github.com/aleju/imgaug) augmentation. For example, passing imgaug.augmenters.Fliplr(0.5) flips images right/left 50% of the time. You can pass complex augmentations as well. This augmentation applies 50% of the time, and when it does it flips images right/left half the time and adds a Gaussian blur with a random sigma in range 0 to 5. augmentation = imgaug.augmenters.Sometimes(0.5, [ imgaug.augmenters.Fliplr(0.5), imgaug.augmenters.GaussianBlur(sigma=(0.0, 5.0)) ]) custom_callbacks: Optional. Add custom callbacks to be called with the keras fit_generator method. Must be list of type keras.callbacks. no_augmentation_sources: Optional. List of sources to exclude for augmentation. A source is string that identifies a dataset and is defined in the Dataset class. rzCreate model in training mode.z(mrcnn\_.*)|(rpn\_.*)|(fpn\_.*)zR(res3.*)|(bn3.*)|(res4.*)|(bn4.*)|(res5.*)|(bn5.*)|(mrcnn\_.*)|(rpn\_.*)|(fpn\_.*)zA(res4.*)|(bn4.*)|(res5.*)|(bn5.*)|(mrcnn\_.*)|(rpn\_.*)|(fpn\_.*)z0(res5.*)|(bn5.*)|(mrcnn\_.*)|(rpn\_.*)|(fpn\_.*)z.*)Zheadsz3+z4+z5+allT)rrgrr)rrrF)rZhistogram_freqZ write_graphZ write_images)rZsave_weights_onlyz Starting at epoch {}. LR={} zCheckpoint Path: {}ntd) Z initial_epochepochsZsteps_per_epoch callbacksZvalidation_dataZvalidation_stepsZmax_queue_sizeworkersZuse_multiprocessingN)rkeysrr/rrrexistsrmakedirsrrZ TensorBoardZModelCheckpointrrrrrrZLEARNING_MOMENTUMr7multiprocessing cpu_countrZ fit_generatorZSTEPS_PER_EPOCHZVALIDATION_STEPSr )rZ train_datasetZ val_datasetrrrrgZcustom_callbacksrrZtrain_generatorZ val_generatorrrrrrtrain> sl+      zMaskRCNN.trainc Csg}g}g}|D]}tj||jj|jj|jj|jjd\}}}} } t||j}td|j |j ||t j |jj gt j d} |||||| qt |}t |}t |}|||fS)a Takes a list of images and modifies them to the format expected as an input to the neural network. images: List of image matrices [height,width,depth]. Images can have different sizes. Returns 3 Numpy matrices: molded_images: [N, h, w, 3]. Images resized and normalized. image_metas: [N, length of meta data]. Details about each image. windows: [N, (y1, x1, y2, x2)]. The portion of the image that has the original image (padding excluded). rLrr)rrSr/rTrUrVrWrrcr r-r_rqrrrx) rrM molded_images image_metaswindowsrZ molded_imagerrjr>rkrrrr mold_inputs s6        zMaskRCNN.mold_inputscCs"t|dddfdkd}|jddkr4|dn|jd}|d|ddf}|d|dftj} |d|df} |t|dddd| f} t||dd}|\} } }}t| | | | g}|| }|| }t||||g}t |||}t ||dd}t|dddf|dddf|dddf|dddfdkd}|jddkrtj ||dd}tj | |dd} tj | |dd} tj | |dd} | jd}g}t |D]&}t | ||||}||q|rtj|d dnt|ddd }|| | |fS) a[Reformats the detections of one image from the format of the neural network output to a format suitable for use in the rest of the application. detections: [N, (y1, x1, y2, x2, class_id, score)] in normalized coordinates mrcnn_mask: [N, height, width, num_classes] original_image_shape: [H, W, C] Original image shape before resizing image_shape: [H, W, C] Shape of the image after resizing and padding window: [y1, x1, y2, x2] Pixel coordinates of box in the image where the real image is excluding the padding. Returns: boxes: [N, (y1, x1, y2, x2)] Bounding boxes in pixels class_ids: [N] Integer class IDs for each bounding box scores: [N] Float probability scores of the class_id masks: [height, width, num_instances] Instance masks NrerrfrTrZr$rr)r)r-rr r\rror norm_boxesrdivideZ denorm_boxesdeleterjZ unmold_maskrrxempty)rrr3original_image_shaper)rZzero_ixrryr rrrrrrshiftZwhZwwrjZ exclude_ixZ full_masksrpZ full_maskrrrunmold_detections sD  B  zMaskRCNN.unmold_detectionsc Cs`|jdksJdt||jjks*Jd|dkrXtdt||D]}td|qH||\}}}|dj}|dd D]}|j|ks~Jd q~||} t | |jjf| j} |dkrtd |td |td | |j j ||| gdd\} } } } } } } g} t |D]L\}}|| || ||j||j||\}}}}| ||||dq| S)aRuns the detection pipeline. images: List of images, potentially of different sizes. Returns a list of dicts, one dict per image. The dict contains: rois: [N, (y1, x1, y2, x2)] detection bounding boxes class_ids: [N] int class IDs scores: [N] float probability scores for the class IDs masks: [H, W, N] instance binary masks Modified by Ondrej Pesek: verbose -> verbosity verbose form in case of verbosity == 3, not 1 rCreate model in inference mode.z'len(images) must be equal to BATCH_SIZErZProcessing {} imagesrrr$Nz\After resizing, all images must have the same size. Check IMAGE_RESIZE_MODE and image sizes.rrrrrr rr)rrwr/rrrrr rr-rrpredictrrr)rrM verbosityrrrrr)grrrr3resultsrp final_roisfinal_class_ids final_scores final_masksrrrdetect* s^       zMaskRCNN.detectcCsZ|jdksJdt||jjks*Jd|rTtdt||D]}td|qD|dj}|ddD]}|j|ksjJd qj||}t ||jjf|j}|rtd |td |td ||j j |||gdd \}} } } } } } g} t |D]^\} }dd|jd|jdg} | || | | |j|| j| \}}}}| ||||dq| S)a8Runs the detection pipeline, but expect inputs that are molded already. Used mostly for debugging and inspecting the model. molded_images: List of images loaded using load_image_gt() image_metas: image meta data, also returned by load_image_gt() Returns a list of dicts, one dict per image. The dict contains: rois: [N, (y1, x1, y2, x2)] detection bounding boxes class_ids: [N] int class IDs scores: [N] float probability scores for the class IDs masks: [H, W, N] instance binary masks rrz,Number of images must be equal to BATCH_SIZErrrr$NzImages must have the same sizerrrrr)rrwr/rrrr rr-rrrrrr)rrrrrr)r rrrr3r rprr r rrrrr detect_moldedu sX       zMaskRCNN.detect_moldedcCs~t|j|}t|dsi|_t||jvrpt|jj|jj||jj |jj }||_ t ||dd|jt|<|jt|S)z0Returns anchor pyramid for the given image size. _anchor_cacheNrT) r0r/rrrrrrrr.rrr)rr)rr^rrrr s     zMaskRCNN.get_anchorscCs|dur |ng}t|dkr dSt|tr.)rrvaluesrZuses_learning_phaserr*Zlearning_phaser%Zfunctionrrr rr-rr/rrrxritemsr)rrMrrrrrZkfrrr)rZmodel_inZ outputs_nprrrrr run_graph s,      zMaskRCNN.run_graph)FN)Nrr$)N)NNN)r)r)N)N)rrrr rrrrrrrrrrrrrrrrrrrrrrrs0 D! 06 * 3 y-F K F   rcCs6t|gt|t|t||gt|}|S)aTakes attributes of an image and puts them in one 1D array. image_id: An int ID of the image. Useful for debugging. original_image_shape: [H, W, C] before resizing or padding. image_shape: [H, W, C] after resizing and padding window: (y1, x1, y2, x2) in pixels. The area of the image where the real image is (excluding the padding) scale: The scaling factor applied to the original image (float32) active_class_ids: List of class_ids available in the dataset from which the image came. Useful if training on images from multiple datasets where not all classes are present in all datasets. )r-rr)rerr)rrjrFmetarrrrc= srccCs|dddf}|ddddf}|ddddf}|ddddf}|dddf}|ddddf}|tj|tj|tj|tj|tj|tjdS) zParses an array that contains image attributes to its components. See compose_image_meta() for more details. meta: [batch, meta length] where meta length depends on NUM_CLASSES Returns a dict of the parsed values. Nrr$rer\ rerr)rrjrF)r\r-rrrrerr)rrjrFrrrparse_image_metaW s      r#cCs|dddf}|ddddf}|ddddf}|ddddf}|dddf}|ddddf}||||||dS) zParses a tensor that contains image attributes to its components. See compose_image_meta() for more details. meta: [batch, meta length] where meta length depends on NUM_CLASSES Returns a dict of the parsed tensors. Nrr$rer\rr r!rr"rrrro srcCs|tj|jS)zExpects an RGB image (or array of images) and subtracts the mean pixel and converts it to float. Expects image colors in RGB order. )r\r-r MEAN_PIXEL)rMr/rrrr srcCs||jtjS)z>Takes a image normalized with mold() and returns the original.)r$r\r-r])Znormalized_imagesr/rrr unmold_image sr% trim_zeroscCs6ttjt|ddtj}tj|||d}||fS)zOften boxes are represented with matrices of shape [N, 4] and are padded with zeros. This removes zero boxes. boxes: [N, 4] matrix of boxes. non_zeros: [N] a 1D boolean mask identifying the rows to keep r$rr9)rwrrEr5r^r)ryr7rrrrr srcCs:g}t|D]}|||d||fq tj|ddS)z_Picks different number of values from each row in x depending on the values in counts. Nrr)rjrrwr)rQZcountsZnum_rowsrrprrrrB s rBcCsXtt|tjd\}}tj||||gddtd}tgd}t|||S)aeConverts boxes from pixel coordinates to normalized coordinates. boxes: [..., (y1, x1, y2, x2)] in pixel coordinates shape: [..., (height, width)] in pixels Note: In pixel coordinates (y2, x2) is outside the box. But in normalized coordinates it's inside the box. Returns: [..., (y1, x1, y2, x2)] in normalized coordinates rTrrrr<r<rr)rwrrrrrrryr rrrjrrrrr s  rcCshtt|tjd\}}tj||||gddtd}tgd}ttt|||tjS)aeConverts boxes from normalized coordinates to pixel coordinates. boxes: [..., (y1, x1, y2, x2)] in normalized coordinates shape: [..., (height, width)] in pixels Note: In pixel coordinates (y2, x2) is outside the box. But in normalized coordinates it's inside the box. Returns: [..., (y1, x1, y2, x2)] in pixel coordinates rTrrrr') rwrrrrrrZmultiplyrr(rrrdenorm_boxes_graph s  r))N)TT)rSTT)FT)Trd)T)FNF)TFNrr$FN)r&)Jr rrYrrr&rX collectionsrrZnumpyr-Z tensorflowrwrZ keras.backendZbackendr*Z keras.layersrrBrZengineZKEZ keras.modelsrr&rZdistutils.versionr __version__rZBatchNormalizationrr0rRrXrsrrZLayerrrrrrrrrr#r(r2r4r9r@rCrGrIrKrlrurzr}rrrcr#rrr%rrBrr)rrrrs         / ? 5aw De39" E ? $"/ w#nI L