python - Problem with getting reproducible results, set seed Tensorflow object detection API

Question

I am using object detection API with tensorflow v1.12. I am having troubles getting reproducible results - each time I run my code I am getting different results. Is there any way to set random seed at training / prediction level? I tried to set seed in model_main.py, but it didn't help.

def main(unused_argv):
    tf.random.set_random_seed(1234)
    flags.mark_flag_as_required('model_dir')
    flags.mark_flag_as_required('pipeline_config_path')
    config = tf.estimator.RunConfig(model_dir=FLAGS.model_dir, tf_random_seed=1234)

My pipeline.config for reference:

model {
  faster_rcnn {
    num_classes: 1
    image_resizer {
      keep_aspect_ratio_resizer {
        min_dimension: 600
        max_dimension: 1024
      }
    }
    feature_extractor {
      type: "faster_rcnn_resnet101"
      first_stage_features_stride: 16
    }
    first_stage_anchor_generator {
      grid_anchor_generator {
        height_stride: 16
        width_stride: 16
        scales: 0.25
        scales: 0.5
        scales: 1.0
        scales: 2.0
        aspect_ratios: 0.5
        aspect_ratios: 1.0
        aspect_ratios: 2.0
      }
    }
    first_stage_box_predictor_conv_hyperparams {
      op: CONV
      regularizer {
        l2_regularizer {
          weight: 0.0
        }
      }
      initializer {
        truncated_normal_initializer {
          stddev: 0.0099999998
        }
      }
    }
    first_stage_nms_score_threshold: 0.0
    first_stage_nms_iou_threshold: 0.69999999
    first_stage_max_proposals: 100
    first_stage_localization_loss_weight: 2.0
    first_stage_objectness_loss_weight: 1.0
    initial_crop_size: 14
    maxpool_kernel_size: 2
    maxpool_stride: 2
    second_stage_box_predictor {
      mask_rcnn_box_predictor {
        fc_hyperparams {
          op: FC
          regularizer {
            l2_regularizer {
              weight: 0.0
            }
          }
          initializer {
            variance_scaling_initializer {
              factor: 1.0
              uniform: true
              mode: FAN_AVG
            }
          }
        }
        use_dropout: false
        dropout_keep_probability: 1.0
      }
    }
    second_stage_post_processing {
      batch_non_max_suppression {
        score_threshold: 0.30000001
        iou_threshold: 0.60000002
        max_detections_per_class: 100
        max_total_detections: 100
      }
      score_converter: SOFTMAX
    }
    second_stage_localization_loss_weight: 2.0
    second_stage_classification_loss_weight: 1.0
  }
}
train_config {
  batch_size: 1
  data_augmentation_options {
    random_horizontal_flip {
    }
  }
  optimizer {
    momentum_optimizer {
      learning_rate {
        manual_step_learning_rate {
          initial_learning_rate: 0.00030000001
          schedule {
            step: 814096
            learning_rate: 2.9999999e-05
          }
        }
      }
      momentum_optimizer_value: 0.89999998
    }
    use_moving_average: false
  }
  gradient_clipping_by_norm: 10.0
  fine_tune_checkpoint: "/home/kombajn/tensorflow/models/research/object_detection/2020_20_11/model_to_send/model.ckpt"
  from_detection_checkpoint: false
  num_steps: 30000
}
train_input_reader {
  label_map_path: "/home/kombajn/tensorflow/models/research/object_detection/2020_22_12_test/data/pack.pbtxt"
  tf_record_input_reader {
    input_path: "/home/kombajn/tensorflow/models/research/object_detection/2020_22_12_test/data/train.record"
  }
}
eval_config {
  num_examples: 100
  use_moving_averages: false
}
eval_input_reader {
  label_map_path: "/home/kombajn/tensorflow/models/research/object_detection/2020_22_12_test/data/pack.pbtxt"
  shuffle: false
  num_readers: 1
  tf_record_input_reader {
    input_path: "/home/kombajn/tensorflow/models/research/object_detection/2020_22_12_test/data/eval.record"
  }
}

Answer 1

Getting repeatable results in tensorflow is a very difficult problem. If you search on Stack Overflow you will find numerous questions on this issue. Bottom line is you have to track down and seed EVERY source of randomness that ran be present either in your model or in the way you generate the data pipeline. This is no easy task. For example weight initialization, generators that shuffle data, dropout layer etc.