treeDepth-19-07-distributedTraining-64-grayscale.log

Starting the distributed training at Sun 21 Jul 23:09:28 BST 2024
Command running in screen session distributed_train_0_20240721-230928 on gpu35
Command running in screen session distributed_train_1_20240721-230928 on gpu36
Command running in screen session distributed_train_2_20240721-230928 on gpu12
Command running in screen session distributed_train_3_20240721-230928 on gpu07
Command running in screen session distributed_train_4_20240721-230928 on gpu08
Command running in screen session distributed_train_5_20240721-230928 on gpu09
Command running in screen session distributed_train_6_20240721-230928 on gpu20
Command running in screen session distributed_train_7_20240721-230928 on gpu02
Log file path is /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/treeDepth-19-07-distributedTraining-64-grayscale.log
Learning rate is now 1e-5 nto 1e-4
Distributed training completed at Sun 21 Jul 23:09:43 BST 2024
[07/21/2024 23:10:00 gluefactory INFO] Starting experiment sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache
[07/21/2024 23:09:57 gluefactory INFO] Restoring from previous training of sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache
[07/21/2024 23:09:57 gluefactory INFO] Restoring from checkpoint checkpoint_best.tar
[07/21/2024 23:09:57 gluefactory INFO] Training in distributed mode with 8 GPUs
[07/21/2024 23:10:04 gluefactory INFO] Starting experiment sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache
[07/21/2024 23:10:17 gluefactory INFO] Restoring from previous training of sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache
[07/21/2024 23:10:17 gluefactory INFO] Restoring from checkpoint checkpoint_best.tar
[07/21/2024 23:10:17 gluefactory INFO] Training in distributed mode with 8 GPUs
[07/21/2024 23:10:17 gluefactory INFO] Using device cuda:0
[07/21/2024 23:10:17 gluefactory.datasets.base_dataset INFO] Creating dataset TreeDepth
[07/21/2024 23:10:17 gluefactory.datasets.treedepth INFO] Initialized TreeDepth dataset with configuration: {'name': 'treedepth', 'num_workers': 1, 'train_batch_size': '???', 'val_batch_size': '???', 'test_batch_size': '???', 'shuffle_training': False, 'batch_size': 8, 'num_threads': 1, 'seed': 0, 'prefetch_factor': 2, 'data_dir': 'syntheticForestData/', 'depth_subpath': 'depthData/', 'image_subpath': 'imageData/', 'info_dir': 'fileLists', 'train_split': 'train_scenes_clean.txt', 'train_num_per_scene': 300, 'val_split': 'valid_scenes_clean.txt', 'val_num_per_scene': None, 'val_pairs': 'valid_pairs.txt', 'test_split': 'test_scenes_clean.txt', 'test_num_per_scene': None, 'test_pairs': None, 'views': 2, 'min_overlap': 0.1, 'max_overlap': 0.7, 'num_overlap_bins': 3, 'sort_by_overlap': False, 'triplet_enforce_overlap': False, 'read_depth': True, 'read_image': True, 'grayscale': False, 'preprocessing': {'resize': None, 'edge_divisible_by': None, 'side': 'long', 'interpolation': 'bilinear', 'align_corners': None, 'antialias': True, 'square_pad': False, 'add_padding_mask': False}, 'p_rotate': 0.0, 'reseed': False, 'load_features': {'do': True, 'path': 'exports/treedepth-undist-depth-r1024_SP-open-k2048-nms3/{scene}.h5', 'data_keys': None, 'device': None, 'trainable': False, 'add_data_path': True, 'collate': False, 'scale': ['keypoints', 'lines', 'orig_lines'], 'padding_fn': 'pad_local_features', 'padding_length': 2048, 'numeric_type': 'float32'}}
[07/21/2024 23:10:19 gluefactory.datasets.treedepth INFO] Sampling new items for train with seed 0.
[07/21/2024 23:10:21 gluefactory.datasets.treedepth INFO] Sampling new items for val with seed 0.
[07/21/2024 23:10:21 gluefactory I[07/21/2024 23:10:27 gluefactory INFO] Parameters with scaled learning rate:
{}
[07/21/2024 23:10:[07/21/2024 23:10:27 gluefactory INFO] Parameters with scaled learning rate:
{}
[07/21/2024 23:10:27 gluefactory INFO] Training with mixed_precision=None
[07/21/2024 23:10:27 gluefactory INFO] Starting training with configuration:
data:
  name: treedepth
  train_split: train_scenes_clean.txt
  train_num_per_scene: 300
  val_split: valid_scenes_clean.txt
  val_pairs: valid_pairs.txt
  min_overlap: 0.1
  max_ovexports/treedepth-undist-depth-r1024_SP-open-k2048-nms3/{scene}.h5
    padding_length: 2048
    padding_fn: pad_local_features
model:
  name: two_view_pipeline
  extractor:
    name: extractors.superpoint_open
    max_num_keypoints: 2048
    force_num_keypoints: true
    detection_threshold: -1
    nms_radius: 3
    trainable: false
  ground_truth:
    name: matchers.depth_matcher
    th_positive: 3
    th_negative: 5
    th_epi: 5
  matcher:
    name: matchers.lightglue
    filter_threshold: 0.1
    flash: false
    checkpointed: true
  allow_no_extract: true
train:
  seed: 0
  epochs: 40
  optimizer: adam
  opt_regexp: null
  optimizer_options: {}
  lr: 1.0e-05
  lr_schedule:
    type: exp
    start: 30
    exp_div_10: 10
    on_epoch: true
    factor: 1.0
    options: {}
  lr_scaling:
  - - 100
    - - dampingnet.const
  eval_every_iter: 500
  save_every_iter: 5000
  log_every_iter: 100
  log_grad_every_iter: null
  test_every_epoch: 1
  keep_last_checkpoints: 10
  load_experiment: pretrain_lightglue
  median_metrics: []
  recall_metrics: {}
  pr_metrics: {}
  best_key: loss/total
  dataset_callback_fn: sample_new_items
  dataset_callback_on_val: false
  clip_grad: 1.0
  pr_curves: {}
  plot:
  - 5
  - gluefactory.visualization.visualize_batch.make_match_figures
  submodules: []
benchmarks:
  treeEval1:
    data:
      preprocessing:
        side: long
        resize: 1600
    eval:
      estimator: opencv
      ransac_th: 0.5
image_mode: null

[07/21/2024 23:10:27 gluefactory INFO] Starting epoch 1
/homes/tp4618/Documents/bitbucket/miniconda3/envs/GlueFactory/lib/python3.10/site-packages/torch/optim/lr_scheduler.py:143: UserWarning: Detected call of `lr_scheduler.step()` before `optimizer.step()`. In PyTorch 1.1.0 and later, you should call them in the opposite order: `optimizer.step()` before `lr_scheduler.step()`.  Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
  warnings.warn("Detected call of `lr_scheduler.step()` before `optimizer.step()`. "
[07/21/2024 23:10:27 gluefactory INFO] lr changed from 1e-05 to 1e-05
[07/21/2024 23:10:27 gluefactory.datasets.treedepth INFO] Sampling new items for train with seed 1.
[07/21/2024 23:10:28 gluefactory INFO] Starting training with configuration:
data:
  name: treedepth
  train_split: train_scenes_clean.txt
  train_num_per_scene: 300
  val_split: valid_scenes_clean.txt
  val_pairs: valid_pairs.txt
  min_overlap: 0.1
  max_overlap: 0.7
  num_overlap_bins: 3
  read_depth: true
  read_image: true
  batch_size: 64
  num_workers: 4
  load_features:
    do: true
    path: exports/treedepth-undist-depth-r1024_SP-open-k2048-nms3/{scene}.h5
    padding_length: 2048
    padding_fn: pad_local_features
model:
  name: two_view_pipeline
  extractor:
    name: extractors.superpoint_open
    max_num_keypoints: 2048
    force_num_keypoints: true
    detection_threshold: -1
    nms_radius: 3
    trainable: false
  ground_truth:
    name: matchers.depth_matcher
    th_positive: 3
    th_negative: 5
    th_epi: 5
  matcher:
    name: matchers.lightglue
    filter_threshold: 0.1
    flash: false
    checkpointed: true
  allow_no_extract: true
train:
  seed: 0
  epochs: 40
  optimizer: adam
  opt_regexp: null
  optimizer_options: {}
  lr: 1.0e-05
  lr_schedule:
    type: exp
    start: 30
    exp_div_10: 10
    on_epoch: true
    factor: 1.0
    options: {}
  lr_scaling:
  - - 100
    - - dampingnet.const
  eval_every_iter: 500
  save_every_iter: 5000
  log_every_iter: 100
  log_grad_every_iter: null
  test_every_epoch: 1
  keep_last_checkpoints: 10
  load_experiment: pretrain_lightglue
  median_metrics: []
  recall_metrics: {}
  pr_metrics: {}
  best_key: loss/total
  dataset_callback_fn: sample_new_items
  dataset_callback_on_val: false
  clip_grad: 1.0
  pr_curves: {}
  plot:
  - 5
  - gluefactory.visualization.visualize_batch.make_match_figures
  submodules: []
benchmarks:
  treeEval1:
    data:
      preprocessing:
        side: long
        resize: 16rices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices


scene_lists_path: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/gluefactory/datasets/tartanSceneLists(Full)


root and info /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/data/syntheticForestData fileLists
!!!!!sample_new_items: num_per_scene: None
205 tools.py utils the classes are  [('TwoViewPipeline', <class 'gluefactory.models.two_view_pipeline.TwoViewPipeline'>)]
ModuleNotFoundError: extractors.superpoint_open
205 tools.py utils the classes are  [('SuperPoint', <class 'gluefactory.models.extractors.superpoint_open.SuperPoint'>)]
None



Superpoint grayscale mode selected
ModuleNotFoundError: matchers.lightglue
205 tools.py utils the classes are  []



 if Path(conf.weights).exists() 368 lightglue loading weights: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/pretrain_lightglue/superpoint_lightglue.pth
state dict lightlue 385 9 to replace keys in range
remvoing log assingments
ModuleNotFoundError: matchers.depth_matcher
205 tools.py utils the classes are  [('DepthMatcher', <class 'gluefactory.models.matchers.depth_matcher.DepthMatcher'>)]
initcp not none
args distibuted
device ids are [device(type='cuda', index=0)]
WARNING: args restore used, using loaded optimizer and scheduler - line 494 train.py
!!!!!sample_new_items: num_per_scene: 300
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
/homes/tp4618/Documents/bitbucket/miniconda3/envs/GlueFactory/lib/python3.10/site-packages/torch/utils/checkpoint.py:464: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.
  warnings.warn(
erPoint'>)]
None



Superpoint grayscale mode selected
ModuleNotFoundError: matchers.lightglue
205 tools.py utils the classes are  []



 if Path(conf.weights).exists() 368 lightglue loading weights: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/pretrain_lightglue/superpoint_lightglue.pth
state dict lightlue 385 9 to replace keys in range
remvoing log assingments
ModuleNotFoundError: matchers.depth_matcher
205 tools.py utils the classes are  [('DepthMatcher', <class 'gluefactory.models.matchers.depth_matcher.DepthMatcher'>)]
initcp not none
args distibuted
device ids are [device(type='cuda', index=0)]
WARNING: args restore used, using loaded optimizer and scheduler - line 494 train.py
!!!!!sample_new_items: num_per_scene: 300
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
ces


scene_lists_path: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/gluefactory/datasets/tartanSceneLists(Full)


root and info /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/data/syntheticForestData fileLists
!!!!!sample_new_items: num_per_scene: None
205 tools.py utils the classes are  [('TwoViewPipeline', <class 'gluefactory.models.two_view_pipeline.TwoViewPipeline'>)]
ModuleNotFoundError: extractors.superpoint_open
205 tools.py utils the classes are  [('SuperPoint', <class 'gluefactory.models.extractors.superpoint_open.SuperPoint'>)]
None



Superpoint grayscale mode selected
ModuleNotFoundError: matchers.lightglue
205 tools.py utils the classes are  []



 if Path(conf.weights).exists() 368 lightglue loading weights: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/pretrain_lightglue/superpoint_lightglue.pth
state dict lightlue 385 9 to replace keys in range
remvoing log assingments
ModuleNotFoundError: matchers.depth_matcher
205 tools.py utils the classes are  [('DepthMatcher', <class 'gluefactory.models.matchers.depth_matcher.DepthMatcher'>)]
initcp not none
args distibuted
device ids are [device(type='cuda', index=0)]
WARNING: args restore used, using loaded optimizer and scheduler - line 494 train.py
!!!!!sample_new_items: num_per_scene: 300
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
/homes/tp4618/Documents/bitbucket/miniconda3/envs/GlueFactory/lib/python3.10/site-packages/torch/utils/checkpoint.py:464: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.
  warnings.warn(
[07/21/2024 23:11:15 gluefactory INFO] [E 1 | it 0] loss {total 6.292E-01, last 5.067E-01, assignment_nll 5.067E-01, nll_pos 8.964E-01, nll_neg 1.169E-01, num_matchable 5.625E+00, num_unmatchable 1.631E+02, confidence 1.491E-02, row_norm 5.607E-02}
/homes/tp4618/Documents/bitbucket/miniconda3/envs/GlueFactory/lib/python3.10/site-packages/torch/utils/checkpoint.py:464: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.
  warnings.warn(
[07/21/2024 23:28:11 gluefactory INFO] [E 1 | it 100] loss {total 6.508E-01, last 5.675E-01, assignment_nll 5.675E-01, nll_pos 1.026E+00, nll_neg 1.094E-01, num_matchable 1.283E+01, num_unmatchable 1.530E+02, confidence 1.027E-02, row_norm 5.543E-02}
[07/21/2024 23:42:49 gluefactory INFO] Intermediate training DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_intermediate_1_200_20240721_234249.pkl
[07/21/2024 23:42:49 gluefactory INFO] [E 1 | it 200] loss {total 3.886E+00, last 3.221E+00, assignment_nll 3.221E+00, nll_pos 5.800E+00, nll_neg 6.420E-01, num_matchable 8.605E+01, num_unmatchable 9.270E+02, confidence 6.308E-02, row_norm 3.449E-01}
[07/21/2024 23:56:06 gluefactory INFO] [E 1 | it 300] loss {total 6.934E-01, last 5.894E-01, assignment_nll 5.894E-01, nll_pos 1.068E+00, nll_neg 1.106E-01, num_matchable 1.262E+01, num_unmatchable 1.552E+02, confidence 1.233E-02, row_norm 5.907E-02}
debugging in data_to_log, iteraiton: 0
Data to be logged to training DataFrame: {'epoch': 1, 'iteration': 0, 'total': 1.3030612468719482, 'last': 1.0676705837249756, 'assignment_nll': 1.0676705837249756, 'nll_pos': 1.9184404611587524, 'nll_neg': 0.2169005423784256, 'num_matchable': 30.765625, 'num_unmatchable': 301.96875, 'confidence': 0.029816627502441406, 'row_norm': 0.11583488434553146}
debugging in data_to_log, iteraiton: 100
Data to be logged to training DataFrame: {'epoch': 1, 'iteration': 100, 'total': 1.325901746749878, 'last': 1.1352753639221191, 'assignment_nll': 1.1352753639221191, 'nll_pos': 2.055079460144043, 'nll_neg': 0.21547120809555054, 'num_matchable': 20.84375, 'num_unmatchable': 324.8203125, 'confidence': 0.020290451124310493, 'row_norm': 0.11609270423650742}
debugging in data_to_log, iteraiton: 200
Data to be logged to training DataFrame: {'epoch': 1, 'iteration': 200, 'total': 1.3470330238342285, 'last': 1.1228381395339966, 'assignment_nll': 1.1228381395339966, 'nll_pos': 2.0401830673217773, 'nll_neg': 0.2054930180311203, 'num_matchable': 34.015625, 'num_unmatchable': 308.8671875, 'confidence': 0.020245913416147232, 'row_norm': 0.11689524352550507}
debugging in data_to_log, iteraiton: 300
Data to be logged to training DataFrame: {'epoch': 1, 'iteration': 300, 'total': 1.3862717151641846, 'last': 1.1952329874038696, 'assignment_nll': 1.1952329874038696, 'nll_pos': 2.1807382106781006, 'nll_neg': 0.20972760021686554, 'num_matchable': 19.171875, 'num_unmatchable': 317.0625, 'confidence': 0.021624907851219177, 'row_norm': 0.12377515435218811}
[07/22/2024 00:10:41 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_1_20000.tar
[07/22/2024 00:35:55 gluefactory INFO] Intermediate training DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_intermediate_1_400_20240722_003555.pkl
[07/22/2024 00:35:55 gluefactory INFO] [E 1 | it 400] loss {total 1.253E+00, last 1.040E+00, assignment_nll 1.040E+00, nll_pos 1.872E+00, nll_neg 2.078E-01, num_matchable 4.227E+01, num_unmatchable 2.844E+02, confidence 2.967E-02, row_norm 1.196E-01}
[07/22/2024 00:46:28 gluefactory INFO] [E 1 | it 500] loss {total 1.236E+00, last 1.042E+00, assignment_nll 1.042E+00, nll_pos 1.888E+00, nll_neg 1.955E-01, num_matchable 2.509E+01, num_unmatchable 3.218E+02, confidence 2.111E-02, row_norm 1.278E-01}
debugging in data_to_log, iteraiton: 400
Data to be logged to training DataFrame: {'epoch': 1, 'iteration': 400, 'total': 1.2533154487609863, 'last': 1.0397942066192627, 'assignment_nll': 1.0397942066192627, 'nll_pos': 1.8717730045318604, 'nll_neg': 0.20781557261943817, 'num_matchable': 42.265625, 'num_unmatchable': 284.4140625, 'confidence': 0.02966853603720665, 'row_norm': 0.11955703794956207}
debugging in data_to_log, iteraiton: 500
Data to be logged to training DataFrame: {'epoch': 1, 'iteration': 500, 'total': 1.2362382411956787, 'last': 1.041847825050354, 'assignment_nll': 1.041847825050354, 'nll_pos': 1.8881933689117432, 'nll_neg': 0.19550232589244843, 'num_matchable': 25.09375, 'num_unmatchable': 321.78125, 'confidence': 0.021112319082021713, 'row_norm': 0.1278170943260193}
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/train.py:922: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  results_df = pd.concat([results_df, new_entry], ignore_index=True)
[07/22/2024 00:54:49 gluefactory INFO] Intermediate DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_20240722_005449.pkl
[07/22/2024 00:54:49 gluefactory INFO] [Validation] {match_recall 4.780E-02, match_precision 1.778E-01, accuracy 8.218E-01, average_precision 2.297E-02, loss/total 3.956E+00, loss/last 3.956E+00, loss/assignment_nll 3.956E+00, loss/nll_pos 7.075E+00, loss/nll_neg 8.383E-01, loss/num_matchable 2.200E+02, loss/num_unmatchable 1.090E+03, loss/row_norm 4.626E-01}
[07/22/2024 00:54:49 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_best.tar
[07/22/2024 00:54:51 gluefactory INFO] New best val: loss/total=3.9564189011864057
/homes/tp4618/Documents/bitbucket/miniconda3/envs/GlueFactory/lib/python3.10/site-packages/torch/utils/checkpoint.py:464: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.
  warnings.warn(
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/train.py:922: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  results_df = pd.concat([results_df, new_entry], ignore_index=True)
[07/22/2024 00:58:59 gluefactory INFO] Intermediate DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_20240722_005859.pkl
[07/22/2024 00:58:59 gluefactory INFO] [Validation] {match_recall 4.780E-02, match_precision 1.778E-01, accuracy 8.218E-01, average_precision 2.297E-02, loss/total 3.956E+00, loss/last 3.956E+00, loss/assignment_nll 3.956E+00, loss/nll_pos 7.074E+00, loss/nll_neg 8.383E-01, loss/num_matchable 2.200E+02, loss/num_unmatchable 1.090E+03, loss/row_norm 4.626E-01}
[07/22/2024 00:58:59 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_best.tar
[07/22/2024 00:59:01 gluefactory INFO] New best val: loss/total=3.956407254140199
/homes/tp4618/Documents/bitbucket/miniconda3/envs/GlueFactory/lib/python3.10/site-packages/torch/utils/checkpoint.py:464: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.
  warnings.warn(
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/train.py:922: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  results_df = pd.concat([results_df, new_entry], ignore_index=True)
[07/22/2024 00:59:34 gluefactory INFO] Intermediate DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_20240722_005934.pkl
[07/22/2024 00:59:34 gluefactory INFO] [Validation] {match_recall 4.780E-02, match_precision 1.778E-01, accuracy 8.218E-01, average_precision 2.297E-02, loss/total 3.956E+00, loss/last 3.956E+00, loss/assignment_nll 3.956E+00, loss/nll_pos 7.074E+00, loss/nll_neg 8.383E-01, loss/num_matchable 2.200E+02, loss/num_unmatchable 1.090E+03, loss/row_norm 4.626E-01}
[07/22/2024 00:59:34 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_best.tar
[07/22/2024 00:59:36 gluefactory INFO] New best val: loss/total=3.956407254140199
/homes/tp4618/Documents/bitbucket/miniconda3/envs/GlueFactory/lib/python3.10/site-packages/torch/utils/checkpoint.py:464: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.
  warnings.warn(
[07/22/2024 01:15:57 gluefactory INFO] Intermediate training DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_intermediate_1_600_20240722_011557.pkl
[07/22/2024 01:15:57 gluefactory INFO] [E 1 | it 600] loss {total 2.139E+00, last 1.753E+00, assignment_nll 1.753E+00, nll_pos 3.133E+00, nll_neg 3.737E-01, num_matchable 4.877E+01, num_unmatchable 3.433E+02, confidence 1.679E-02, row_norm 1.351E-01}
[07/22/2024 01:25:12 gluefactory INFO] [E 1 | it 700] loss {total 1.166E+00, last 9.734E-01, assignment_nll 9.734E-01, nll_pos 1.755E+00, nll_neg 1.920E-01, num_matchable 2.162E+01, num_unmatchable 3.280E+02, confidence 2.174E-02, row_norm 1.326E-01}
[07/22/2024 01:34:25 gluefactory INFO] Intermediate training DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_intermediate_1_800_20240722_013425.pkl
[07/22/2024 01:34:25 gluefactory INFO] [E 1 | it 800] loss {total 1.136E+00, last 9.595E-01, assignment_nll 9.595E-01, nll_pos 1.716E+00, nll_neg 2.029E-01, num_matchable 3.006E+01, num_unmatchable 3.277E+02, confidence 1.568E-02, row_norm 1.222E-01}
[07/22/2024 01:43:40 gluefactory INFO] [E 1 | it 900] loss {total 1.354E+00, last 1.092E+00, assignment_nll 1.092E+00, nll_pos 1.994E+00, nll_neg 1.910E-01, num_matchable 2.230E+01, num_unmatchable 3.220E+02, confidence 2.099E-02, row_norm 1.272E-01}
[07/22/2024 01:50:03 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_1_25000.tar
[07/22/2024 01:52:56 gluefactory INFO] Intermediate training DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_intermediate_1_1000_20240722_015256.pkl
[07/22/2024 01:52:56 gluefactory INFO] [E 1 | it 1000] loss {total 1.419E+00, last 1.211E+00, assignment_nll 1.211E+00, nll_pos 2.210E+00, nll_neg 2.113E-01, num_matchable 1.944E+01, num_unmatchable 3.282E+02, confidence 2.054E-02, row_norm 1.271E-01}
Data to be logged: {'epoch': 1, 'iteration': 500, 'match_recall': 0.047799176436033854, 'match_precision': 0.17775924090065626, 'accuracy': 0.8218046702399959, 'average_precision': 0.022967494261669056, 'loss/total': 3.9564189011864057, 'loss/last': 3.9564189011864057, 'loss/assignment_nll': 3.9564189011864057, 'loss/nll_pos': 7.07451172441581, 'loss/nll_neg': 0.8383260773756213, 'loss/num_matchable': 220.0270450011084, 'loss/num_unmatchable': 1090.2701174905785, 'loss/row_norm': 0.46263143345564967}
debugging in data_to_log, iteraiton: 600
Data to be logged to training DataFrame: {'epoch': 1, 'iteration': 600, 'total': 1.111169695854187, 'last': 0.9115743637084961, 'assignment_nll': 0.9115743637084961, 'nll_pos': 1.6364758014678955, 'nll_neg': 0.18667274713516235, 'num_matchable': 17.171875, 'num_unmatchable': 343.265625, 'confidence': 0.016788482666015625, 'row_norm': 0.13507160544395447}
debugging in data_to_log, iteraiton: 700
Data to be logged to training DataFrame: {'epoch': 1, 'iteration': 700, 'total': 1.1663577556610107, 'last': 0.973437488079071, 'assignment_nll': 0.973437488079071, 'nll_pos': 1.7548638582229614, 'nll_neg': 0.19201114773750305, 'num_matchable': 21.625, 'num_unmatchable': 328.0390625, 'confidence': 0.02173580974340439, 'row_norm': 0.1326490342617035}
debugging in data_to_log, iteraiton: 800
Data to be logged to training DataFrame: {'epoch': 1, 'iteration': 800, 'total': 1.1358617544174194, 'last': 0.9594579935073853, 'assignment_nll': 0.9594579935073853, 'nll_pos': 1.7159721851348877, 'nll_neg': 0.20294366776943207, 'num_matchable': 30.0625, 'num_unmatchable': 327.6640625, 'confidence': 0.01567729189991951, 'row_norm': 0.12224555015563965}
debugging in data_to_log, iteraiton: 900
Data to be logged to training DataFrame: {'epoch': 1, 'iteration': 900, 'total': 1.3540093898773193, 'last': 1.0923504829406738, 'assignment_nll': 1.0923504829406738, 'nll_pos': 1.9936652183532715, 'nll_neg': 0.19103561341762543, 'num_matchable': 22.296875, 'num_unmatchable': 321.9609375, 'confidence': 0.020985186100006104, 'row_norm': 0.127243310213089}
debugging in data_to_log, iteraiton: 1000
Data to be logged to training DataFrame: {'epoch': 1, 'iteration': 1000, 'total': 1.4193062782287598, 'last': 1.2105036973953247, 'assignment_nll': 1.2105036973953247, 'nll_pos': 2.20973801612854, 'nll_neg': 0.2112693190574646, 'num_matchable': 19.4375, 'num_un: 981.5234375, 'confidence': 0.06087867170572281, 'row_norm': 0.3858395516872406}
[07/22/2024 02:12:54 gluefactory INFO] Intermediate DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_20240722_021254.pkl
[07/22/2024 02:12:54 gluefactory INFO] [Validation] {match_recall 6.014E-02, match_precision 1.938E-01, accuracy 8.240E-01, average_precision 2.832E-02, loss/total 3.885E+00, loss/last 3.885E+00, loss/assignment_nll 3.885E+00, loss/nll_pos 6.969E+00, loss/nll_neg 8.018E-01, loss/num_matchable 2.200E+02, loss/num_unmatchable 1.090E+03, loss/row_norm 4.791E-01}
[07/22/2024 02:12:54 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_best.tar
[07/22/2024 02:12:56 gluefactory INFO] New best val: loss/total=3.8853865893340065
[07/22/2024 02:23:02 gluefactory INFO] [E 1 | it 1100] loss {total 3.554E+00, last 2.819E+00, assignment_nll 2.819E+00, nll_pos 5.087E+00, nll_neg 5.515E-01, num_matchable 8.909E+01, num_unmatchable 9.185E+02, confidence 8.630E-02, row_norm 3.937E-01}
[07/22/2024 02:32:17 gluefactory INFO] Intermediate training DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_intermediate_1_1200_20240722_023217.pkl
[07/22/2024 02:32:17 gluefactory INFO] [E 1 | it 1200] loss {total 3.648E+00, last 3.020E+00, assignment_nll 3.020E+00, nll_pos 5.435E+00, nll_neg 6.045E-01, num_matchable 7.614E+01, num_unmatchable 9.824E+02, confidence 8.548E-02, row_norm 3.794E-01}
[07/22/2024 02:41:31 gluefactory INFO] [E 1 | it 1300] loss {total 3.223E+00, last 2.554E+00, assignment_nll 2.554E+00, nll_pos 4.509E+00, nll_neg 5.991E-01, num_matchable 1.086E+02, num_unmatchable 9.386E+02, confidence 1.050E-01, row_norm 3.651E-01}
[07/22/2024 02:50:46 gluefactory INFO] Intermediate training DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_intermediate_1_1400_20240722_025046.pkl
[07/22/2024 02:50:46 gluefactory INFO] [E 1 | it 1400] loss {total 3.597E+00, last 2.970E+00, assignment_nll 2.970E+00, nll_pos 5.343E+00, nll_neg 5.980E-01, num_matchable 7.533E+01, num_unmatchable 9.698E+02, confidence 8.443E-02, row_norm 3.827E-01}
[07/22/2024 03:00:00 gluefactory INFO] [E 1 | it 1500] loss {total 1.338E+00, last 1.141E+00, assignment_nll 1.141E+00, nll_pos 2.074E+00, nll_neg 2.074E-01, num_matchable 1.458E+01, num_unmatchable 3.280E+02, confidence 2.011E-02, row_norm 1.197E-01}
Data to be logged: {'epoch': 1, 'iteration': 1000, 'match_recall': 0.06014162442182704, 'match_precision': 0.19381281771170436, 'accuracy': 0.824012804655034, 'average_precision': 0.028317508098240025, 'loss/total': 3.88539784717285, 'loss/last': 3.88539784717285, 'loss/assignment_nll': 3.88539784717285, 'loss/nll_pos': 6.969012823424089, 'loss/nll_neg': 0.8017828479306816, 'loss/num_matchable': 220.0270450011084, 'loss/num_unmatchable': 1090.2701174905785, 'loss/row_norm': 0.4790702485583675}
debugging in data_to_log, iteraiton: 1100
Data to be logged to training DataFrame: {'epoch': 1, 'iteration': 1100, 'total': 1.1494877338409424, 'last': 0.900067925453186, 'assignment_nll': 0.900067925453186, 'nll_pos': 1.624417781829834, 'nll_neg': 0.17571808397769928, 'num_matchable': 31.96875, 'num_unmatchable': 301.796875, '.734375, 'confidence': 0.047597452998161316, 'row_norm': 0.19905292987823486}
debugging in data_to_log, iteraiton: 1200
Data to be logged to training DataFrame: {'epoch': 1, 'iteration': 1200, 'total': 1.8015711307525635, 'last': 1.4633047580718994, 'assignment_nll': 1.4633047580718994, 'nll_pos': 2.6175010204315186, 'nll_neg': 0.30910858511924744, 'num_matchable': 43.078125, 'num_unmatchable': 489.0625, 'confidence': 0.04986748471856117, 'row_norm': 0.1870894730091095}
debugging in data_to_log, iteraiton: 1300
Data to be logged to training DataFrame: {'epoch': 1, 'iteration': 1300, 'total': 1.4664115905761719, 'last': 1.20814049243927, 'assignment_nll': 1.20814049243927, 'nll_pos': 2.1111607551574707, 'nll_neg': 0.3051202893257141, 'num_matchable': 28.484375, 'num_unmatchable': 506.71875, 'confidence': 0.03205902501940727, 'row_norm': 0.18552353978157043}
debugging in data_to_log, iteraiton: 1400
Data to be logged to training DataFrame: {'epoch': 1, 'iteration': 1400, 'total': 1.7748818397521973, 'last': 1.451672911643982, 'assignment_nll': 1.451672911643982, 'nll_pos': 2.6013457775115967, 'nll_neg': 0.3020000457763672, 'num_matchable': 47.734375, 'num_unmatchable': 460.109375, 'confidence': 0.04970063269138336, 'row_norm': 0.18910843133926392}
debugging in data_to_log, iteraiton: 1500
Data to be logged to training DataFrame: {'epoch': 1, 'iteration': 1500, 'total': 2.021202564239502, 'last': 1.725255012512207, 'assignment_nll': 1.725255012512207, 'nll_pos': 3.1428189277648926, 'nll_neg': 0.30769115686416626, 'num_matchable': 26.125, 'num_unmatchable': 479.8828125, 'confidence': 0.03294437378644943, 'row_norm': 0.18166008591651917}
[07/22/2024 03:13:16 gluefactory INFO] Intermediate DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_20240722_031316.pkl
[07/22/2024 03:13:16 gluefactory INFO] [Validation] {match_recall 7.118E-02, match_precision 2.049E-01, accuracy 8.234E-01, average_precision 3.207E-02, loss/total 3.771E+00, loss/last 3.771E+00, loss/assignment_nll 3.771E+00, loss/nll_pos 6.754E+00, loss/nll_neg 7.887E-01, loss/num_matchable 2.200E+02, loss/num_unmatchable 1.090E+03, loss/row_norm 4.812E-01}
[07/22/2024 03:13:16 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_best.tar
[07/22/2024 03:13:28 gluefactory INFO] New best val: loss/total=3.771481891292554
[07/22/2024 03:30:42 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_1_30000.tar
[07/22/2024 03:31:17 gluefactory INFO] Intermediate training DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_intermediate_1_1600_20240722_033117.pkl
[07/22/2024 03:31:17 gluefactory INFO] [E 1 | it 1600] loss {total 1.742E+00, last 1.440E+00, assignment_nll 1.440E+00, nll_pos 2.591E+00, nll_neg 2.897E-01, num_matchable 4.012E+01, num_unmatchable 4.934E+02, confidence 3.106E-02, row_norm 1.913E-01}
[07/22/2024 03:40:30 gluefactory INFO] [E 1 | it 1700] loss {total 1.634E+00, last 1.354E+00, assignment_nll 1.354E+00, nll_pos 2.426E+00, nll_neg 2.812E-01, num_matchable 2.989E+01, num_unmatchable 4.971E+02, confidence 3.094E-02, row_norm 1.981E-01}
[07/22/2024 03:49:44 gluefactory INFO] Intermediate training DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_intermediate_1_1800_20240722_034944.pkl
[07/22/2024 03:49:44 gluefactory INFO] [E 1 | it 1800] loss {total 1.605E+00, last 1.250E+00, assignment_nll 1.250E+00, nll_pos 2.213E+00, nll_neg 2.874E-01, num_matchable 3.831E+01, num_unmatchable 4.921E+02, confidence 4.268E-02, row_norm 1.992E-01}
[07/22/2024 03:58:58 gluefactory INFO] [E 1 | it 1900] loss {total 1.142E+00, last 9.223E-01, assignment_nll 9.223E-01, nll_pos 1.639E+00, nll_neg 2.062E-01, num_matchable 4.002E+01, num_unmatchable 2.862E+02, confidence 3.969E-02, row_norm 1.232E-01}
[07/22/2024 04:08:12 gluefactory INFO] Intermediate training DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_intermediate_1_2000_20240722_040812.pkl
[07/22/2024 04:08:12 gluefactory INFO] [E 1 | it 2000] loss {total 4.727E+00, last 3.857E+00, assignment_nll 3.857E+00, nll_pos 6.909E+00, nll_neg 8.064E-01, num_matchable 8.369E+01, num_unmatchable 1.324E+03, confidence 9.226E-02, row_norm 5.037E-01}
Data to be logged: {'epoch': 1, 'iteration': 1500, 'match_recall': 0.0711811681134976, 'match_precision': 0.2048857039147445, 'accuracy': 0.8233931083104161, 'average_precision': 0.032070884822073056, 'loss/total': 3.7715173086341007, 'loss/last': 3.7715173086341007, 'loss/assignment_nll': 3.7715173086341007, 'loss/nll_pos': 6.754315921087483, 'loss/nll_neg': 0.7887186758324249, 'loss/num_matchable': 220.02771004211925, 'loss/num_unmatchable': 1090.2707825315895, 'loss/row_norm': 0.48120508265849415}
debugging in data_to_log, iteraiton: 1600
Data to be logged to training DataFrame: {'epoch': 1, 'iteration': 1600, 'total': 4.721128463745117, 'last': 3.811270236968994, 'assignment_nll': 3.811270236968994, 'nll_pos': 6.844436168670654, 'nll_neg': 0.7781034708023071, 'num_matchable': 138.5, 'num_unmatchable': 1251.2734375, 'confidence': 0.10980042815208435, 'row_norm': 0.505553662776947}
debugging in data_to_log, iteraiton: 1700
Data to be logged to training DataFrame: {'epoch': 1, 'iteration': 1700, 'total': 4.399725437164307, 'last': 3.555422782897949, 'assignment_nll': 3.555422782897949, 'nll_pos': 6.391109943389893, 'nll_neg': 0.7197356820106506, 'num_matchable': 82.015625, 'num_unmatchable': 1328.828125, 'confidence': 0.09547221660614014, 'row_norm': 0.536541759967804}
debugging in data_to_log, iteraiton: 1800
Data to be logged to training DataFrame: {'epoch': 1, 'iteration': 1800, 'total': 4.5227131843566895, 'last': 3.5291738510131836, 'assignment_nll': 3.5291738510131836, 'nll_pos': 6.264069080352783, 'nll_neg': 0.7942790985107422, 'num_matchable': 108.40625, 'num_unmatchable': 1287.375, 'confidence': 0.10841839015483856, 'row_norm': 0.5082392692565918}
debugging in data_to_log, iteraiton: 1900
Data to be logged to training DataFrame: {'epoch': 1, 'iteration': 1900, 'total': 4.409704208374023, 'last': 3.513228416442871, 'assignment_nll': 3.513228416442871, 'nll_pos': 6.232780456542969, 'nll_neg': 0.7936770915985107, 'num_matchable': 114.3125, 'num_unmatchable': 1265.9375, 'confidence': 0.13276301324367523, 'row_norm': 0.5080801844596863}
debugging in data_to_log, iteraiton: 2000
Data to be logged to training DataFrame: {'epoch': 1, 'iteration': 2000, 'total': 4.727288246154785, 'last': 3.857477903366089, 'assignment_nll': 3.857477903366089, 'nll_pos': 6.908600807189941, 'nll_neg': 0.8063543438911438, 'num_matchable': 83.6875, 'num_unmatchable': 1324.2890625, 'confidence': 0.09226229041814804, 'row_norm': 0.5036670565605164}
[07/22/2024 04:17:43 gluefactory INFO] Intermediate DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_20240722_041743.pkl
[07/22/2024 04:17:43 gluefactory INFO] [Validation] {match_recall 8.010E-02, match_precision 2.097E-01, accuracy 8.236E-01, average_precision 3.537E-02, loss/total 3.721E+00, loss/last 3.721E+00, loss/assignment_nll 3.721E+00, loss/nll_pos 6.614E+00, loss/nll_neg 8.282E-01, loss/num_matchable 2.200E+02, loss/num_unmatchable 1.090E+03, loss/row_norm 4.711E-01}
[07/22/2024 04:17:43 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_best.tar
[07/22/2024 04:17:44 gluefactory INFO] New best val: loss/total=3.7211035570107867
[07/22/2024 04:37:31 gluefactory INFO] [E 1 | it 2100] loss {total 4.714E+00, last 3.756E+00, assignment_nll 3.756E+00, nll_pos 6.756E+00, nll_neg 7.566E-01, num_matchable 9.272E+01, num_unmatchable 1.314E+03, confidence 1.338E-01, row_norm 5.297E-01}
Data to be logged: {'epoch': 1, 'iteration': 2000, 'match_recall': 0.0801018406016613, 'match_precision': 0.2097287496567975, 'accuracy': 0.8236286032361061, 'average_precision': 0.035370445544472225, 'loss/total': 3.7210659129316754, 'loss/last': 3.7210659129316754, 'loss/assignment_nll': 3.7210659129316754, 'loss/nll_pos': 6.613930830958677, 'loss/nll_neg': 0.828200977146163, 'loss/num_matchable': 220.0266016404345, 'loss/num_unmatchable': 1090.2698958102417, 'loss/row_norm': 0.4710895361907729}
debugging in data_to_log, iteraiton: 2100
Data to be logged to training DataFrame: {'epoch': 1, 'iteration': 2100, 'total': 0.6197258234024048, 'last': 0.5180568099021912, 'assignment_nll': 0.5180568099021912, 'nll_pos': 0.9379504323005676, 'nll_neg': 0.09816314280033112, 'num_matchable': 13.578125, 'num_unmatchable': 149.6171875, 'confidence': 0.018979784101247787, 'row_norm': 0.0651213675737381}
[07/22/2024 04:50:56 gluefactory INFO] Intermediate DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_20240722_045056.pkl
[07/22/2024 04:50:56 gluefactory INFO] [Validation] {match_recall 9.494E-02, match_precision 2.109E-01, accuracy 8.213E-01, average_precision 3.986E-02, loss/total 3.683E+00, loss/last 3.683E+00, loss/assignment_nll 3.683E+00, loss/nll_pos 6.586E+00, loss/nll_neg 7.805E-01, loss/num_matchable 2.200E+02, loss/num_unmatchable 1.090E+03, loss/row_norm 4.959E-01}
[07/22/2024 04:50:56 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_best.tar
[07/22/2024 04:50:57 gluefactory INFO] New best val: loss/total=3.6833882925912618
[07/22/2024 04:51:00 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_1_34488.tar
[07/22/2024 04:51:01 gluefactory INFO] Starting epoch 2
[07/22/2024 04:51:01 gluefactory INFO] lr changed from 1e-05 to 1e-05
[07/22/2024 04:51:01 gluefactory.datasets.treedepth INFO] Sampling new items for train with seed 2.
Data to be logged: {'epoch': 1, 'iteration': 2155, 'match_recall': 0.09494294721973436, 'match_precision': 0.21087476300256092, 'accuracy': 0.8212955625906564, 'average_precision': 0.039864311599314924, 'loss/total': 3.6833882925912618, 'loss/last': 3.6833882925912618, 'loss/assignment_nll': 3.6833882925912618, 'loss/nll_pos': 6.586239023447512, 'loss/nll_neg': 0.7805375502131198, 'loss/num_matchable': 220.0270450011084, 'loss/num_unmatchable': 1090.2701174905785, 'loss/row_norm': 0.4959058904298999}
!!!!!sample_new_items: num_per_scene: 300
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
[07/22/2024 04:51:56 gluefactory INFO] Intermediate DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_20240722_045156.pkl
[07/22/2024 04:51:56 gluefactory INFO] [Validation] {match_recall 9.494E-02, match_precision 2.109E-01, accuracy 8.213E-01, average_precision 3.986E-02, loss/total 3.683E+00, loss/last 3.683E+00, loss/assignment_nll 3.683E+00, loss/nll_pos 6.586E+00, loss/nll_neg 7.805E-01, loss/num_matchable 2.200E+02, loss/num_unmatchable 1.090E+03, loss/row_norm 4.959E-01}
[07/22/2024 04:51:56 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_best.tar
[07/22/2024 04:51:57 gluefactory INFO] New best val: loss/total=3.68341194666349
[07/22/2024 04:52:00 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_1_34488.tar
[07/22/2024 04:52:02 gluefactory INFO] Starting epoch 2
[07/22/2024 04:52:02 gluefactory INFO] lr changed from 1e-05 to 1e-05
[07/22/2024 04:52:02 gluefactory.datasets.treedepth INFO] Sampling new items for train with seed 2.
Data to be logged: {'epoch': 1, 'iteration': 2155, 'match_recall': 0.09494205550706653, 'match_precision': 0.21087548637118936, 'accuracy': 0.8212959078245643, 'average_precision': 0.039864225209494314, 'loss/total': 3.68341194666349, 'loss/last': 3.68341194666349, 'loss/assignment_nll': 3.68341194666349, 'loss/nll_pos': 6.58628551649746, 'loss/nll_neg': 0.7805384004546808, 'loss/num_matchable': 220.02771004211925, 'loss/num_unmatchable': 1090.2707825315895, 'loss/row_norm': 0.49590589307253546}
!!!!!sample_new_items: num_per_scene: 300
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
[07/22/2024 04:55:08 gluefactory INFO] Intermediate DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_20240722_045508.pkl
[07/22/2024 04:55:08 gluefactory INFO] [Validation] {match_recall 9.494E-02, match_precision 2.109E-01, accuracy 8.213E-01, average_precision 3.986E-02, loss/total 3.683E+00, loss/last 3.683E+00, loss/assignment_nll 3.683E+00, loss/nll_pos 6.586E+00, loss/nll_neg 7.805E-01, loss/num_matchable 2.200E+02, loss/num_unmatchable 1.090E+03, loss/row_norm 4.959E-01}
[07/22/2024 04:55:08 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_best.tar
[07/22/2024 04:55:10 gluefactory INFO] New best val: loss/total=3.6833781057597528
[07/22/2024 04:55:14 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_1_34488.tar
[07/22/2024 04:55:15 gluefactory INFO] Starting epoch 2
[07/22/2024 04:55:15 gluefactory INFO] lr changed from 1e-05 to 1e-05
[07/22/2024 04:55:15 gluefactory.datasets.treedepth INFO] Sampling new items for train with seed 2.
Data to be logged: {'epoch': 1, 'iteration': 2155, 'match_recall': 0.09494336948161679, 'match_precision': 0.21087476300256092, 'accuracy': 0.8212955168202086, 'average_precision': 0.03986464400653057, 'loss/total': 3.6833781057597528, 'loss/last': 3.6833781057597528, 'loss/assignment_nll': 3.6833781057597528, 'loss/nll_pos': 6.58621934997721, 'loss/nll_neg': 0.7805368630750247, 'loss/num_matchable': 220.0266016404345, 'loss/num_unmatchable': 1090.2698958102417, 'loss/row_norm': 0.4959058904298999}
!!!!!sample_new_items: num_per_scene: 300
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
[07/22/2024 04:55:42 gluefactory INFO] Intermediate DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_20240722_045542.pkl
[07/22/2024 04:55:42 gluefactory INFO] [Validation] {match_recall 9.494E-02, match_precision 2.109E-01, accuracy 8.213E-01, average_precision 3.986E-02, loss/total 3.683E+00, loss/last 3.683E+00, loss/assignment_nll 3.683E+00, loss/nll_pos 6.586E+00, loss/nll_neg 7.805E-01, loss/num_matchable 2.200E+02, loss/num_unmatchable 1.090E+03, loss/row_norm 4.959E-01}
[07/22/2024 04:55:42 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_best.tar
[07/22/2024 04:55:43 gluefactory INFO] New best val: loss/total=3.6833781057597528
[07/22/2024 04:55:47 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_1_34488.tar
[07/22/2024 04:55:49 gluefactory INFO] Starting epoch 2
[07/22/2024 04:55:49 gluefactory INFO] lr changed from 1e-05 to 1e-05
[07/22/2024 04:55:49 gluefactory.datasets.treedepth INFO] Sampling new items for train with seed 2.
Data to be logged: {'epoch': 1, 'iteration': 2155, 'match_recall': 0.09494336948161679, 'match_precision': 0.21087476300256092, 'accuracy': 0.8212955168202086, 'average_precision': 0.03986464400653057, 'loss/total': 3.6833781057597528, 'loss/last': 3.6833781057597528, 'loss/assignment_nll': 3.6833781057597528, 'loss/nll_pos': 6.58621934997721, 'loss/nll_neg': 0.7805368630750247, 'loss/num_matchable': 220.0266016404345, 'loss/num_unmatchable': 1090.2698958102417, 'loss/row_norm': 0.4959058904298999}
!!!!!sample_new_items: num_per_scene: 300
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
[07/22/2024 04:58:55 gluefactory INFO] Intermediate DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_20240722_045855.pkl
[07/22/2024 04:58:55 gluefactory INFO] [Validation] {match_recall 9.494E-02, match_precision 2.109E-01, accuracy 8.213E-01, average_precision 3.986E-02, loss/total 3.683E+00, loss/last 3.683E+00, loss/assignment_nll 3.683E+00, loss/nll_pos 6.586E+00, loss/nll_neg 7.805E-01, loss/num_matchable 2.200E+02, loss/num_unmatchable 1.090E+03, loss/row_norm 4.959E-01}
[07/22/2024 04:58:55 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_best.tar
[07/22/2024 04:58:57 gluefactory INFO] New best val: loss/total=3.6833781057597528
[07/22/2024 04:59:04 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_1_34488.tar
[07/22/2024 04:59:05 gluefactory INFO] Starting epoch 2
[07/22/2024 04:59:05 gluefactory INFO] lr changed from 1e-05 to 1e-05
[07/22/2024 04:59:05 gluefactory.datasets.treedepth INFO] Sampling new items for train with seed 2.
Data to be logged: {'epoch': 1, 'iteration': 2155, 'match_recall': 0.09494336948161679, 'match_precision': 0.21087476300256092, 'accuracy': 0.8212955168202086, 'average_precision': 0.03986464400653057, 'loss/total': 3.6833781057597528, 'loss/last': 3.6833781057597528, 'loss/assignment_nll': 3.6833781057597528, 'loss/nll_pos': 6.58621934997721, 'loss/nll_neg': 0.7805368630750247, 'loss/num_matchable': 220.0266016404345, 'loss/num_unmatchable': 1090.2698958102417, 'loss/row_norm': 0.4959058904298999}
!!!!!sample_new_items: num_per_scene: 300
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
[07/22/2024 05:02:31 gluefactory INFO] Intermediate DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_20240722_050231.pkl
[07/22/2024 05:02:31 gluefactory INFO] [Validation] {match_recall 9.494E-02, match_precision 2.109E-01, accuracy 8.213E-01, average_precision 3.986E-02, loss/total 3.683E+00, loss/last 3.683E+00, loss/assignment_nll 3.683E+00, loss/nll_pos 6.586E+00, loss/nll_neg 7.805E-01, loss/num_matchable 2.200E+02, loss/num_unmatchable 1.090E+03, loss/row_norm 4.959E-01}
[07/22/2024 05:02:31 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_best.tar
[07/22/2024 05:02:33 gluefactory INFO] New best val: loss/total=3.6833781057597528
[07/22/2024 05:02:37 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_1_34488.tar
[07/22/2024 05:02:38 gluefactory INFO] Starting epoch 2
[07/22/2024 05:02:38 gluefactory INFO] lr changed from 1e-05 to 1e-05
[07/22/2024 05:02:38 gluefactory.datasets.treedepth INFO] Sampling new items for train with seed 2.
Data to be logged: {'epoch': 1, 'iteration': 2155, 'match_recall': 0.09494336948161679, 'match_precision': 0.21087476300256092, 'accuracy': 0.8212955168202086, 'average_precision': 0.03986464400653057, 'loss/total': 3.6833781057597528, 'loss/last': 3.6833781057597528, 'loss/assignment_nll': 3.6833781057597528, 'loss/nll_pos': 6.58621934997721, 'loss/nll_neg': 0.7805368630750247, 'loss/num_matchable': 220.0266016404345, 'loss/num_unmatchable': 1090.2698958102417, 'loss/row_norm': 0.4959058904298999}
!!!!!sample_new_items: num_per_scene: 300
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
[07/22/2024 05:03:04 gluefactory INFO] [E 2 | it 0] loss {total 3.469E+00, last 2.725E+00, assignment_nll 2.725E+00, nll_pos 4.880E+00, nll_neg 5.691E-01, num_matchable 6.400E+01, num_unmatchable 9.796E+02, confidence 9.658E-02, row_norm 4.010E-01}
[07/22/2024 05:08:53 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_2_35000.tar
[07/22/2024 05:12:20 gluefactory INFO] [E 2 | it 100] loss {total 3.447E+00, last 2.710E+00, assignment_nll 2.710E+00, nll_pos 4.790E+00, nll_neg 6.307E-01, num_matchable 9.183E+01, num_unmatchable 9.288E+02, confidence 9.345E-02, row_norm 3.675E-01}
[07/22/2024 05:21:35 gluefactory INFO] Intermediate training DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_intermediate_2_200_20240722_052135.pkl
[07/22/2024 05:21:35 gluefactory INFO] [E 2 | it 200] loss {total 3.501E+00, last 2.774E+00, assignment_nll 2.774E+00, nll_pos 5.019E+00, nll_neg 5.289E-01, num_matchable 6.797E+01, num_unmatchable 9.782E+02, confidence 8.265E-02, row_norm 3.992E-01}
[07/22/2024 05:30:50 gluefactory INFO] [E 2 | it 300] loss {total 3.174E+00, last 2.462E+00, assignment_nll 2.462E+00, nll_pos 4.354E+00, nll_neg 5.703E-01, num_matchable 9.030E+01, num_unmatchable 9.302E+02, confidence 1.059E-01, row_norm 3.905E-01}
[07/22/2024 05:40:04 gluefactory INFO] Intermediate training DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_intermediate_2_400_20240722_054004.pkl
[07/22/2024 05:40:04 gluefactory INFO] [E 2 | it 400] loss {total 3.297E+00, last 2.545E+00, assignment_nll 2.545E+00, nll_pos 4.531E+00, nll_neg 5.594E-01, num_matchable 7.431E+01, num_unmatchable 9.776E+02, confidence 9.795E-02, row_norm 4.075E-01}
[07/22/2024 05:49:19 gluefactory INFO] [E 2 | it 500] loss {total 1.194E+00, last 9.352E-01, assignment_nll 9.352E-01, nll_pos 1.676E+00, nll_neg 1.944E-01, num_matchable 3.998E+01, num_unmatchable 3.032E+02, confidence 3.370E-02, row_norm 1.248E-01}
debugging in data_to_log, iteraiton: 0
Data to be logged to training DataFrame: {'epoch': 2, 'iteration': 0, 'total': 1.1410539150238037, 'last': 0.927067756652832, 'assignment_nll': 0.927067756652832, 'nll_pos': 1.6738351583480835, 'nll_neg': 0.18030047416687012, 'num_matchable': 22.109375, 'num_unmatchable': 328.046875, 'confidence': 0.03235162794589996, 'row_norm': 0.13837632536888123}
debugging in data_to_log, iteraiton: 100
Data to be logged to training DataFrame: {'epoch': 2, 'iteration': 100, 'total': 1.0685124397277832, 'last': 0.8598470091819763, 'assignl': 2.2500762939453125, 'nll_pos': 3.966240882873535, 'nll_neg': 0.5339117646217346, 'num_matchable': 69.390625, 'num_unmatchable': 778.75, 'confidence': 0.07340928167104721, 'row_norm': 0.3061252236366272}
debugging in data_to_log, iteraiton: 200
Data to be logged to training DataFrame: {'epoch': 2, 'iteration': 200, 'total': 2.822678804397583, 'last': 2.2039742469787598, 'assignment_nll': 2.2039742469787598, 'nll_pos': 3.977308750152588, 'nll_neg': 0.4306398630142212, 'num_matchable': 52.765625, 'num_unmatchable': 823.453125, 'confidence': 0.06791045516729355, 'row_norm': 0.3375470042228699}
debugging in data_to_log, iteraiton: 300
Data to be logged to training DataFrame: {'epoch': 2, 'iteration': 300, 'total': 2.5645577907562256, 'last': 2.0072851181030273, 'assignment_nll': 2.0072851181030273, 'nll_pos': 3.5351786613464355, 'nll_neg': 0.4793914556503296, 'num_matchable': 71.75, 'num_unmatchable': 778.9921875, 'confidence': 0.08635648339986801, 'row_norm': 0.32599765062332153}
debugging in data_to_log, iteraiton: 400
Data to be logged to training DataFrame: {'epoch': 2, 'iteration': 400, 'total': 2.7148613929748535, 'last': 2.0973408222198486, 'assignment_nll': 2.0973408222198486, 'nll_pos': 3.7292985916137695, 'nll_neg': 0.46538305282592773, 'num_matchable': 63.984375, 'num_unmatchable': 804.8359375, 'confidence': 0.07967410236597061, 'row_norm': 0.33827275037765503}
debugging in data_to_log, iteraiton: 500
Data to be logged to training DataFrame: {'epoch': 2, 'iteration': 500, 'total': 2.984821319580078, 'last': 2.3608925342559814, 'assignment_nll': 2.3608925342559814, 'nll_pos': 4.22625207901001, 'nll_neg': 0.4955330491065979, 'num_matchable': 97.796875, 'num_unmatchable': 743.6875, 'confidence': 0.09228193759918213, 'row_norm': 0.3069427013397217}
[07/22/2024 06:02:11 gluefactory INFO] Intermediate DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_20240722_060211.pkl
[07/22/2024 06:02:11 gluefactory INFO] [Validation] {match_recall 1.042E-01, match_precision 2.169E-01, accuracy 8.219E-01, average_precision 4.340E-02, loss/total 3.645E+00, loss/last 3.645E+00, loss/assignment_nll 3.645E+00, loss/nll_pos 6.455E+00, loss/nll_neg 8.349E-01, loss/num_matchable 2.200E+02, loss/num_unmatchable 1.090E+03, loss/row_norm 4.724E-01}
[07/22/2024 06:02:11 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_best.tar
[07/22/2024 06:02:13 gluefactory INFO] New best val: loss/total=3.645123619484653
[07/22/2024 06:18:44 gluefactory INFO] Intermediate training DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_intermediate_2_600_20240722_061844.pkl
[07/22/2024 06:18:44 gluefactory INFO] [E 2 | it 600] loss {total 2.748E+00, last 2.153E+00, assignment_nll 2.153E+00, nll_pos 3.828E+00, nll_neg 4.781E-01, num_matchable 4.350E+01, num_unmatchable 8.403E+02, confidence 6.775E-02, row_norm 3.368E-01}
[07/22/2024 06:26:53 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_2_40000.tar
[07/22/2024 06:28:01 gluefactory INFO] [E 2 | it 700] loss {total 2.870E+00, last 2.237E+00, assignment_nll 2.237E+00, nll_pos 3.996E+00, nll_neg 4.784E-01, num_matchable 7.464E+01, num_unmatchable 7.927E+02, confidence 9.195E-02, row_norm 3.239E-01}
[07/22/2024 06:37:17 gluefactory INFO] Intermediate training DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_intermediate_2_800_20240722_063717.pkl
[07/22/2024 06:37:17 gluefactory INFO] [E 2 | it 800] loss {total 1.189E+00, last 9.166E-01, assignment_nll 9.166E-01, nll_pos 1.645E+00, nll_neg 1.879E-01, num_matchable 2.728E+01, num_unmatchable 3.154E+02, confidence 3.466E-02, row_norm 1.323E-01}
[07/22/2024 06:46:31 gluefactory INFO] [E 2 | it 900] loss {total 1.020E+00, last 7.694E-01, assignment_nll 7.694E-01, nll_pos 1.344E+00, nll_neg 1.948E-01, num_matchable 2.978E+01, num_unmatchable 3.113E+02, confidence 3.962E-02, row_norm 1.285E-01}
[07/22/2024 06:55:45 gluefactory INFO] Intermediate training DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_intermediate_2_1000_20240722_065545.pkl
[07/22/2024 06:55:45 gluefactory INFO] [E 2 | it 1000] loss {total 1.175E+00, last 9.380E-01, assignment_nll 9.380E-01, nll_pos 1.683E+00, nll_neg 1.932E-01, num_matchable 2.528E+01, num_unmatchable 3.194E+02, confidence 4.107E-02, row_norm 1.301E-01}
Data to be logged: {'epoch': 2, 'iteration': 500, 'match_recall': 0.10424833555712658, 'match_precision': 0.2168575654450884, 'accuracy': 0.8219403791945569, 'average_precision': 0.04339691757215881, 'loss/total': 3.645133040163264, 'loss/last': 3.645133040163264, 'loss/assignment_nll': 3.645133040163264, 'loss/nll_pos': 6.455394673416182, 'loss/nll_neg': 0.8348714085487801, 'loss/num_matchable': 220.0270450011084, 'loss/num_unmatchable': 1090.2701174905785, 'loss/row_norm': 0.4724022360657726}
debugging in data_to_log, iteraiton: 600
Data to be logged to training DataFrame: {'epoch': 2, 'iteration': 600, 'total': 0.9737132787704468, 'last': 0.7369191646575928, 'assignment_nll': 0.7369191646575928, 'nll_pos': 1.2828996181488037, 'nll_neg': 0.190938800573349, 'num_matchable': 22.046875, 'num_unmatchable': 329.578125, 'confidence': 0.03331862762570381, 'row_norm': 0.1350584775209427}
debugging in data_to_log, iteraiton: 700
Data to be logged to training DataFrame: {'epoch': 2, 'iteration': 700, 'total': 1.1112661361694336, 'last': 0.8421453833580017, 'assignment_nll': 0.8421453833580017, 'nll_pos': 1.4962830543518066, 'nll_neg': 0.18800772726535797, 'num_matchable': 41.4375, 'num_unmatchable': 293.125, 'confidence': 0.045048654079437256, 'row_norm': 0.12974828481674194}
debugging in data_to_log, iteraiton: 800
Data to be logged to training DataFrame: {'epoch': 2, 'iteration': 800, 'total': 1.189452886581421, 'last': 0.9165714383125305, 'assignment_nll': 0.9165714383125305, 'nll_pos': 1.6452311277389526, 'nll_neg': 0.18791179358959198, 'num_matchable': 27.28125, 'num_unmatchable': 315.3671875, 'confidence': 0.03465929999947548, 'row_norm': 0.13234205543994904}
debugging in data_to_log, iteraiton: 900
Data to be logged to training DataFrame: {'epoch': 2, 'iteration': 900, 'total': 1.019899606704712, 'last': 0.7694402933120728, 'assignment_nll': 0.7694402933120728, 'nll_pos': 1.3441120386123657, 'nll_neg': 0.19476856291294098, 'num_matchable': 29.78125, 'num_unmatchable': 311.2734375, 'confidence': 0.03962039574980736, 'row_norm': 0.12849891185760498}
debugging in data_to_log, iteraiton: 1000
Data to be logged to training DataFrame: {'epoch': 2, 'iteration': 1000, 'total': 1.1752656698226929, 'last': 0.9379969835281372, 'assignment_nll': 0.9379969835281372, 'nll_pos': 1.6828004121780396, 'nll_neg': 0.1931934356689453, 'num_matchable': 25.28125, 'num_unmatchable': 319.359375, 'confidence': 0.041069284081459045, 'row_norm': 0.1301107406616211}
[07/22/2024 07:04:05 gluefactory INFO] Intermediate DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_20240722_070405.pkl
[07/22/2024 07:04:05 gluefactory INFO] [Validation] {match_recall 1.092E-01, match_precision 2.216E-01, accuracy 8.218E-01, average_precision 4.548E-02, loss/total 3.595E+00, loss/last 3.595E+00, loss/assignment_nll 3.595E+00, loss/nll_pos 6.430E+00, loss/nll_neg 7.604E-01, loss/num_matchable 2.200E+02, loss/num_unmatchable 1.090E+03, loss/row_norm 5.059E-01}
[07/22/2024 07:04:05 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_best.tar
[07/22/2024 07:04:06 gluefactory INFO] New best val: loss/total=3.5949363550995015
[07/22/2024 07:24:56 gluefactory INFO] [E 2 | it 1100] loss {total 1.145E+00, last 8.939E-01, assignment_nll 8.939E-01, nll_pos 1.589E+00, nll_neg 1.992E-01, num_matchable 2.045E+01, num_unmatchable 3.170E+02, confidence 3.883E-02, row_norm 1.317E-01}
[07/22/2024 07:34:10 gluefactory INFO] Intermediate training DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_intermediate_2_1200_20240722_073410.pkl
[07/22/2024 07:34:10 gluefactory INFO] [E 2 | it 1200] loss {total 1.854E+00, last 1.532E+00, assignment_nll 1.532E+00, nll_pos 5.782E+00, nll_neg 6.619E-01, num_matchable 8.288E+01, num_unmatchable 1.111E+03, confidence 1.348E-01, row_norm 4.709E-01}
[07/22/2024 07:43:23 gluefactory INFO] [E 2 | it 1300] loss {total 4.013E+00, last 3.114E+00, assignment_nll 3.114E+00, nll_pos 5.589E+00, nll_neg 6.395E-01, num_matchable 8.330E+01, num_unmatchable 1.153E+03, confidence 1.233E-01, row_norm 4.728E-01}
[07/22/2024 07:44:35 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_2_45000.tar
[07/22/2024 07:52:38 gluefactory INFO] Intermediate training DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_intermediate_2_1400_20240722_075238.pkl
[07/22/2024 07:52:38 gluefactory INFO] [E 2 | it 1400] loss {total 3.975E+00, last 3.097E+00, assignment_nll 3.097E+00, nll_pos 5.540E+00, nll_neg 6.547E-01, num_matchable 9.712E+01, num_unmatchable 1.102E+03, confidence 1.341E-01, row_norm 4.704E-01}
[07/22/2024 08:01:52 gluefactory INFO] [E 2 | it 1500] loss {total 4.018E+00, last 3.199E+00, assignment_nll 3.199E+00, nll_pos 5.751E+00, nll_neg 6.464E-01, num_matchable 7.620E+01, num_unmatchable 1.139E+03, confidence 1.219E-01, row_norm 4.655E-01}
Data to be logged: {'epoch': 2, 'iteration': 1000, 'match_recall': 0.10922075230745931, 'match_precision': 0.22164292216203518, 'accuracy': 0.8217687697718187, 'average_precision': 0.045479412852175036, 'loss/total': 3.5949538968614134, 'loss/last': 3.5949538968614134, 'loss/assignment_nll': 3.5949538968614134, 'loss/nll_pos': 6.429555894212801, 'loss/nll_neg': 0.760351893326259, 'loss/num_matchable': 220.02771004211925, 'loss/num_unmatchable': 1090.2707825315895, 'loss/row_norm': 0.5059385270613627}
debugging in data_to_log, iteraiton: 1100
Data to be logged to training DataFrame: {'epoch': 2, 'iteration': 1100, 'total': 4.026790618896484, 'last': 3.1510725021362305, 'assignment_nll': 3.1510725021362305, 'nll_pos': 5.621626377105713, 'nll_neg': 0.6805189847946167, 'num_matchable': 66.296875, 'num_unmatchable': 1135.890625, 'confidence': 0.12262899428606033, 'row_norm': 0.4637126922607422}
debugging in data_to_log, iteraiton: 1200
Data to be logged to training DataFrame: {'epoch': 2, 'iteration': 1200, 'total': 4.061588287353516, 'last': 3.2220046520233154, 'assignment_nll': 3.2220046520233154, 'nll_pos': 5.782106876373291, 'nll_neg': 0.6619031429290771, 'num_matchable': 82.875, 'num_unmatchable': 1111.2890625, 'confidence': 0.13484281301498413, 'row_norm': 0.47085514664649963}
debugging in data_to_log, iteraiton: 1300
Data to be logged to training DataFrame: {'epoch': 2, 'iteration': 1300, 'total': 4.013425827026367, 'last': 3.114142894744873, 'assignment_nll': 3.114142894744873, 'nll_pos': 5.588755130767822, 'nll_neg': 0.6395305395126343, 'num_matchable': 83.296875, 'num_unmatchable': 1152.96875, 'confidence': 0.12325713783502579, 'row_norm': 0.4727685749530792}
debugging in data_to_log, iteraiton: 1400
Data to be logged to training DataFrame: {'epoch': 2, 'iteration': 1400, 'total': 3.9750418663024902, 'last': 3.0971450805664062, 'assignment_nll': 3.0971450805664062, 'nll_pos': 5.5396294593811035, 'nll_neg': 0.6546608805656433, 'num_matchable': 97.125, 'num_unmatchable': 1102.1015625, 'confidence': 0.13407988846302032, 'row_norm': 0.4703793525695801}
debugging in data_to_log, iteraiton: 1500
Data to be logged to training DataFrame: {'epoch': 2, 'iteration': 1500, 'total': 4.017781734466553, 'last': 3.1988284587860107, 'assignment_nll': 3.1988284587860107, 'nll_pos': 5.751286029815674, 'nll_neg': 0.6463708281517029, 'num_matchable': 76.203125, 'num_unmatchable': 1138.8515625, 'confidence': 0.12190749496221542, 'row_norm': 0.46551749110221863}
[07/22/2024 08:09:50 gluefactory INFO] Intermediate DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_20240722_080950.pkl
[07/22/2024 08:09:50 gluefactory INFO] [Validation] {match_recall 1.121E-01, match_precision 2.218E-01, accuracy 8.226E-01, average_precision 4.647E-02, loss/total 3.602E+00, loss/last 3.602E+00, loss/assignment_nll 3.602E+00, loss/nll_pos 6.436E+00, loss/nll_neg 7.685E-01, loss/num_matchable 2.200E+02, loss/num_unmatchable 1.090E+03, loss/row_norm 5.001E-01}
[07/22/2024 08:10:11 gluefactory INFO] Intermediate DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_20240722_081011.pkl
[07/22/2024 08:10:11 gluefactory INFO] [Validation] {match_recall 1.121E-01, match_precision 2.218E-01, accuracy 8.226E-01, average_precision 4.647E-02, loss/total 3.602E+00, loss/last 3.602E+00, loss/assignment_nll 3.602E+00, loss/nll_pos 6.436E+00, loss/nll_neg 7.685E-01, loss/num_matchable 2.200E+02, loss/num_unmatchable 1.090E+03, loss/row_norm 5.001E-01}
[07/22/2024 08:11:03 gluefactory INFO] Intermediate DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_20240722_081103.pkl
[07/22/2024 08:11:03 gluefactory INFO] [Validation] {match_recall 1.121E-01, match_precision 2.218E-01, accuracy 8.226E-01, average_precision 4.647E-02, loss/total 3.602E+00, loss/last 3.602E+00, loss/assignment_nll 3.602E+00, loss/nll_pos 6.436E+00, loss/nll_neg 7.685E-01, loss/num_matchable 2.200E+02, loss/num_unmatchable 1.090E+03, loss/row_norm 5.001E-01}
[07/22/2024 08:14:19 gluefactory INFO] Intermediate DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_20240722_081419.pkl
[07/22/2024 08:14:19 gluefactory INFO] [Validation] {match_recall 1.121E-01, match_precision 2.218E-01, accuracy 8.226E-01, average_precision 4.647E-02, loss/total 3.602E+00, loss/last 3.602E+00, loss/assignment_nll 3.602E+00, loss/nll_pos 6.436E+00, loss/nll_neg 7.685E-01, loss/num_matchable 2.200E+02, loss/num_unmatchable 1.090E+03, loss/row_norm 5.001E-01}
[07/22/2024 08:14:43 gluefactory INFO] Intermediate DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_20240722_081443.pkl
[07/22/2024 08:14:43 gluefactory INFO] [Validation] {match_recall 1.121E-01, match_precision 2.218E-01, accuracy 8.226E-01, average_precision 4.647E-02, loss/total 3.602E+00, loss/last 3.602E+00, loss/assignment_nll 3.602E+00, loss/nll_pos 6.436E+00, loss/nll_neg 7.685E-01, loss/num_matchable 2.200E+02, loss/num_unmatchable 1.090E+03, loss/row_norm 5.001E-01}
[07/22/2024 08:15:01 gluefactory INFO] Intermediate DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_20240722_081501.pkl
[07/22/2024 08:15:01 gluefactory INFO] [Validation] {match_recall 1.121E-01, match_precision 2.218E-01, accuracy 8.226E-01, average_precision 4.647E-02, loss/total 3.602E+00, loss/last 3.602E+00, loss/assignment_nll 3.602E+00, loss/nll_pos 6.436E+00, loss/nll_neg 7.685E-01, loss/num_matchable 2.200E+02, loss/num_unmatchable 1.090E+03, loss/row_norm 5.001E-01}
[07/22/2024 08:15:59 gluefactory INFO] Intermediate DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_20240722_081559.pkl
[07/22/2024 08:15:59 gluefactory INFO] [Validation] {match_recall 1.121E-01, match_precision 2.218E-01, accuracy 8.226E-01, average_precision 4.647E-02, loss/total 3.602E+00, loss/last 3.602E+00, loss/assignment_nll 3.602E+00, loss/nll_pos 6.436E+00, loss/nll_neg 7.685E-01, loss/num_matchable 2.200E+02, loss/num_unmatchable 1.090E+03, loss/row_norm 5.001E-01}
[07/22/2024 08:21:47 gluefactory INFO] Intermediate DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_20240722_082147.pkl
[07/22/2024 08:21:47 gluefactory INFO] [Validation] {match_recall 1.121E-01, match_precision 2.218E-01, accuracy 8.226E-01, average_precision 4.647E-02, loss/total 3.602E+00, loss/last 3.602E+00, loss/assignment_nll 3.602E+00, loss/nll_pos 6.436E+00, loss/nll_neg 7.685E-01, loss/num_matchable 2.200E+02, loss/num_unmatchable 1.090E+03, loss/row_norm 5.001E-01}
[07/22/2024 08:31:00 gluefactory INFO] Intermediate training DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_intermediate_2_1600_20240722_083100.pkl
[07/22/2024 08:31:00 gluefactory INFO] [E 2 | it 1600] loss {total 3.418E+00, last 2.586E+00, assignment_nll 2.586E+00, nll_pos 4.626E+00, nll_neg 5.461E-01, num_matchable 6.536E+01, num_unmatchable 9.654E+02, confidence 8.712E-02, row_norm 4.106E-01}
[07/22/2024 08:40:14 gluefactory INFO] [E 2 | it 1700] loss {total 3.257E+00, last 2.484E+00, assignment_nll 2.484E+00, nll_pos 4.430E+00, nll_neg 5.381E-01, num_matchable 9.370E+01, num_unmatchable 9.507E+02, confidence 1.205E-01, row_norm 4.102E-01}
[07/22/2024 08:49:38 gluefactory INFO] Intermediate training DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_intermediate_2_1800_20240722_084938.pkl
[07/22/2024 08:49:38 gluefactory INFO] [E 2 | it 1800] loss {total 3.223E+00, last 2.520E+00, assignment_nll 2.520E+00, nll_pos 4.521E+00, nll_neg 5.196E-01, num_matchable 6.636E+01, num_unmatchable 9.723E+02, confidence 9.660E-02, row_norm 4.113E-01}
[07/22/2024 08:58:51 gluefactory INFO] [E 2 | it 1900] loss {total 3.413E+00, last 2.638E+00, assignment_nll 2.638E+00, nll_pos 4.720E+00, nll_neg 5.571E-01, num_matchable 7.870E+01, num_unmatchable 9.429E+02, confidence 1.250E-01, row_norm 4.065E-01}
[07/22/2024 09:02:22 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_2_50000.tar
[07/22/2024 09:08:06 gluefactory INFO] Intermediate training DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_intermediate_2_2000_20240722_090806.pkl
[07/22/2024 09:08:06 gluefactory INFO] [E 2 | it 2000] loss {total 4.348E+00, last 3.222E+00, assignment_nll 3.222E+00, nll_pos 5.695E+00, nll_neg 7.486E-01, num_matchable 1.651E+02, num_unmatchable 1.171E+03, confidence 2.244E-01, row_norm 5.271E-01}
Data to be logged: {'epoch': 2, 'iteration': 1500, 'match_recall': 0.11214209860866892, 'match_precision': 0.22181553641679783, 'accuracy': 0.8226183721322068, 'average_precision': 0.046468670546719194, 'loss/total': 3.6021006018537767, 'loss/last': 3.6021006018537767, 'loss/assignment_nll': 3.6021006018537767, 'loss/nll_pos': 6.435713182608745, 'loss/nll_neg': 0.7684880358975681, 'loss/num_matchable': 220.02771004211925, 'loss/num_unmatchable': 1090.2707825315895, 'loss/row_norm': 0.5000701261396499}
debugging in data_to_log, iteraiton: 1600
Data to be logged to training DataFrame: {'epoch': 2, 'iteration': 1600, 'total': 4.331210136413574, 'last': 3.276435613632202, 'assignment_nll': 3.276435613632202, 'nll_pos': 5.84014368057251, 'nll_neg': 0.7127277851104736, 'num_matchable': 93.03125, 'num_unmatchable': 1296.0390625, 'confidence': 0.11905716359615326, 'row_norm': 0.5528572201728821}
debugging in data_to_log, iteraiton: 1700
Data to be logged to training DataFrame: {'epoch': 2, 'iteration': 1700, 'total': 4.434722900390625, 'last': 3.423588275909424, 'assignment_nll': 3.423588275909424, 'nll_pos': 6.113668918609619, 'nll_neg': 0.7335069179534912, 'num_matchable': 120.671875, 'num_unmatchable': 1260.9453125, 'confidence': 0.15239167213439941, 'row_norm': 0.5374276638031006}
debugging in data_to_log, iteraiton: 1800
Data to be logged to training DataFrame: {'epoch': 2, 'iteration': 1800, 'total': 4.388836860656738, 'last': 3.419078826904297, 'assignment_nll': 3.419078826904297, 'nll_pos': 6.13602876663208, 'nll_neg': 0.702128350734707, 'num_matchable': 66.359375, 'num_unmatchable': 972.3203125, 'confidence': 0.09659957885742188, 'row_norm': 0.4112884998321533}
debugging in data_to_log, iteraiton: 1900
Data to be logged to training DataFrame: {'epoch': 2, 'iteration': 1900, 'total': 3.4134280681610107, 'last': 2.6384432315826416, 'assignment_nll': 2.6384432315826416, 'nll_pos': 4.719820976257324, 'nll_neg': 0.5570658445358276, 'num_matchable': 78.703125, 'num_unmatchable': 942.8515625, 'confidence': 0.12495358288288116, 'row_norm': 0.40648263692855835}
debugging in data_to_log, iteraiton: 2000
Data to be logged to training DataFrame: {'epoch': 2, 'iteration': 2000, 'total': 3.200399875640869, 'last': 2.3269400596618652, 'assignment_nll': 2.3269400596618652, 'nll_pos': 4.108447551727295, 'nll_neg': 0.5454323291778564, 'num_matchable': 134.796875, 'num_unmatchable': 862.984375, 'confidence': 0.17058700323104858, 'row_norm': 0.3972740173339844}
[07/22/2024 09:28:09 gluefactory INFO] Intermediate DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_20240722_092750.pkl
[07/22/2024 09:27:50 gluefactory INFO] [Validation] {match_recall 1.299E-01, match_precision 2.281E-01, accuracy 8.201E-01, average_precision 5.201E-02, loss/total 3.518E+00, loss/last 3.518E+00, loss/assignment_nll 3.518E+00, loss/nll_pos 6.263E+00, loss/nll_neg 7.734E-01, loss/num_matchable 2.200E+02, loss/num_unmatchable 1.090E+03, loss/row_norm 5.030E-01}
[07/22/2024 09:27:50 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_best.tar
[07/22/2024 09:27:52 gluefactory INFO] New best val: loss/total=3.5183872418666833
[07/22/2024 09:37:26 gluefactory INFO] [E 2 | it 2100] loss {total 5.987E-01, last 5.029E-01, assignment_nll 5.029E-01, nll_pos 9.086E-01, nll_neg 9.725E-02, num_matchable 1.347E+01, num_unmatchable 1.565E+02, confidence 1.911E-02, row_norm 6.562E-02}
Data to be logged: {'epoch': 2, 'iteration': 2000, 'match_recall': 0.12990561964970376, 'match_precision': 0.2280802829007343, 'accuracy': 0.8200923562499114, 'average_precision': 0.05201262447698788, 'loss/total': 3.5183872418666833, 'loss/last': 3.5183872418666833, 'loss/assignment_nll': 3.5183872418666833, 'loss/nll_pos': 6.263359536619362, 'loss/nll_neg': 0.7734149155080834, 'loss/num_matchable': 220.0266016404345, 'loss/num_unmatchable': 1090.2698958102417, 'loss/row_norm': 0.5030074648094981}
debugging in data_to_log, iteraiton: 2100
Data to be logged to training DataFrame: {'epoch': 2, 'iteration': 2100, 'total': 0.5986723899841309, 'last': 0.5029388070106506, 'assignment_nll': 0.5029388070106506, 'nll_pos': 0.9086293578147888, 'nll_neg': 0.09724822640419006, 'num_matchable': 13.46875, 'num_unmatchable': 156.5, 'confidence': 0.01910868100821972, 'row_norm': 0.06562460958957672}
5}
}
[07/22/2024 09:50:31 gluefactory INFO] Intermediate DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_20240722_095031.pkl
[07/22/2024 09:50:31 gluefactory INFO] [Validation] {match_recall 1.116E-01, match_precision 2.406E-01, accuracy 8.261E-01, average_precision 4.933E-02, loss/total 3.479E+00, loss/last 3.479E+00, loss/assignment_nll 3.479E+00, loss/nll_pos 6.144E+00, loss/nll_neg 8.139E-01, loss/num_matchable 2.200E+02, loss/num_unmatchable 1.090E+03, loss/row_norm 4.762E-01}
[07/22/2024 09:50:31 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_best.tar
[07/22/2024 09:50:32 gluefactory INFO] New best val: loss/total=3.4789911695394515
[07/22/2024 09:50:34 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_2_51736.tar
[07/22/2024 09:50:36 gluefactory INFO] Starting epoch 3
[07/22/2024 09:50:36 gluefactory INFO] lr changed from 1e-05 to 1e-05
[07/22/2024 09:50:36 gluefactory.datasets.treedepth INFO] Sampling new items for train with seed 3.
Data to be logged: {'epoch': 2, 'iteration': 2155, 'match_recall': 0.11163495837862956, 'match_precision': 0.2406342272743553, 'accuracy': 0.8260505995922727, 'average_precision': 0.04932982456208028, 'loss/total': 3.4789911695394515, 'loss/last': 3.4789911695394515, 'loss/assignment_nll': 3.4789911695394515, 'loss/nll_pos': 6.144101483888992, 'loss/nll_neg': 0.8138808389112759, 'loss/num_matchable': 220.02771004211925, 'loss/num_unmatchable': 1090.2707825315895, 'loss/row_norm': 0.47619561899583057}
!!!!!sample_new_items: num_per_scene: 300
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
[07/22/2024 09:51:44 gluefactory INFO] Intermediate DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_20240722_095144.pkl
[07/22/2024 09:51:44 gluefactory INFO] [Validation] {match_recall 1.116E-01, match_precision 2.406E-01, accuracy 8.261E-01, average_precision 4.933E-02, loss/total 3.479E+00, loss/last 3.479E+00, loss/assignment_nll 3.479E+00, loss/nll_pos 6.144E+00, loss/nll_neg 8.139E-01, loss/num_matchable 2.200E+02, loss/num_unmatchable 1.090E+03, loss/row_norm 4.762E-01}
[07/22/2024 09:51:44 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_best.tar
[07/22/2024 09:51:45 gluefactory INFO] New best val: loss/total=3.4789911695394515
[07/22/2024 09:51:49 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_2_51736.tar
[07/22/2024 09:51:51 gluefactory INFO] Starting epoch 3
[07/22/2024 09:51:51 gluefactory INFO] lr changed from 1e-05 to 1e-05
[07/22/2024 09:51:51 gluefactory.datasets.treedepth INFO] Sampling new items for train with seed 3.
Data to be logged: {'epoch': 2, 'iteration': 2155, 'match_recall': 0.11163495837862956, 'match_precision': 0.2406342272743553, 'accuracy': 0.8260505995922727, 'average_precision': 0.04932982456208028, 'loss/total': 3.4789911695394515, 'loss/last': 3.4789911695394515, 'loss/assignment_nll': 3.4789911695394515, 'loss/nll_pos': 6.144101483888992, 'loss/nll_neg': 0.8138808389112759, 'loss/num_matchable': 220.02771004211925, 'loss/num_unmatchable': 1090.2707825315895, 'loss/row_norm': 0.47619561899583057}
!!!!!sample_new_items: num_per_scene: 300
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
[07/22/2024 09:55:26 gluefactory INFO] Intermediate DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_20240722_095525.pkl
[07/22/2024 09:55:26 gluefactory INFO] [Validation] {match_recall 1.116E-01, match_precision 2.406E-01, accuracy 8.261E-01, average_precision 4.933E-02, loss/total 3.479E+00, loss/last 3.479E+00, loss/assignment_nll 3.479E+00, loss/nll_pos 6.144E+00, loss/nll_neg 8.139E-01, loss/num_matchable 2.200E+02, loss/num_unmatchable 1.090E+03, loss/row_norm 4.762E-01}
[07/22/2024 09:55:26 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_best.tar
[07/22/2024 09:55:27 gluefactory INFO] New best val: loss/total=3.4789610175963808
[07/22/2024 09:55:31 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_2_51736.tar
[07/22/2024 09:55:33 gluefactory INFO] Starting epoch 3
[07/22/2024 09:55:33 gluefactory INFO] lr changed from 1e-05 to 1e-05
[07/22/2024 09:55:33 gluefactory.datasets.treedepth INFO] Sampling new items for train with seed 3.
Data to be logged: {'epoch': 2, 'iteration': 2155, 'match_recall': 0.11163967149394102, 'match_precision': 0.2406418744406106, 'accuracy': 0.8260524337399009, 'average_precision': 0.049332689319507086, 'loss/total': 3.4789610175963808, 'loss/last': 3.4789610175963808, 'loss/assignment_nll': 3.4789610175963808, 'loss/nll_pos': 6.144042911246251, 'loss/nll_neg': 0.8138791353098439, 'loss/num_matchable': 220.0266016404345, 'loss/num_unmatchable': 1090.2698958102417, 'loss/row_norm': 0.4761956669332394}
!!!!!sample_new_items: num_per_scene: 300
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
[07/22/2024 09:56:26 gluefactory INFO] Intermediate DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_20240722_095626.pkl
[07/22/2024 09:56:26 gluefactory INFO] [Validation] {match_recall 1.116E-01, match_precision 2.406E-01, accuracy 8.261E-01, average_precision 4.933E-02, loss/total 3.479E+00, loss/last 3.479E+00, loss/assignment_nll 3.479E+00, loss/nll_pos 6.144E+00, loss/nll_neg 8.139E-01, loss/num_matchable 2.200E+02, loss/num_unmatchable 1.090E+03, loss/row_norm 4.762E-01}
[07/22/2024 09:56:26 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_best.tar
[07/22/2024 09:56:27 gluefactory INFO] New best val: loss/total=3.4789610175963808
[07/22/2024 09:56:34 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_2_51736.tar
[07/22/2024 09:56:36 gluefactory INFO] Starting epoch 3
[07/22/2024 09:56:36 gluefactory INFO] lr changed from 1e-05 to 1e-05
[07/22/2024 09:56:36 gluefactory.datasets.treedepth INFO] Sampling new items for train with seed 3.
Data to be logged: {'epoch': 2, 'iteration': 2155, 'match_recall': 0.11163967149394102, 'match_precision': 0.2406418744406106, 'accuracy': 0.8260524337399009, 'average_precision': 0.049332689319507086, 'loss/total': 3.4789610175963808, 'loss/last': 3.4789610175963808, 'loss/assignment_nll': 3.4789610175963808, 'loss/nll_pos': 6.144042911246251, 'loss/nll_neg': 0.8138791353098439, 'loss/num_matchable': 220.0266016404345, 'loss/num_unmatchable': 1090.2698958102417, 'loss/row_norm': 0.4761956669332394}
!!!!!sample_new_items: num_per_scene: 300
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
[07/22/2024 10:02:25 gluefactory INFO] Intermediate DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_20240722_100225.pkl
[07/22/2024 10:02:25 gluefactory INFO] [Validation] {match_recall 1.116E-01, match_precision 2.406E-01, accuracy 8.261E-01, average_precision 4.933E-02, loss/total 3.479E+00, loss/last 3.479E+00, loss/assignment_nll 3.479E+00, loss/nll_pos 6.144E+00, loss/nll_neg 8.139E-01, loss/num_matchable 2.200E+02, loss/num_unmatchable 1.090E+03, loss/row_norm 4.762E-01}
[07/22/2024 10:02:25 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_best.tar
[07/22/2024 10:02:27 gluefactory INFO] New best val: loss/total=3.4789610175963808
[07/22/2024 10:02:30 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_2_51736.tar
[07/22/2024 10:02:32 gluefactory INFO] Starting epoch 3
[07/22/2024 10:02:32 gluefactory INFO] lr changed from 1e-05 to 1e-05
[07/22/2024 10:02:32 gluefactory.datasets.treedepth INFO] Sampling new items for train with seed 3.
Data to be logged: {'epoch': 2, 'iteration': 2155, 'match_recall': 0.11163967149394102, 'match_precision': 0.2406418744406106, 'accuracy': 0.8260524337399009, 'average_precision': 0.049332689319507086, 'loss/total': 3.4789610175963808, 'loss/last': 3.4789610175963808, 'loss/assignment_nll': 3.4789610175963808, 'loss/nll_pos': 6.144042911246251, 'loss/nll_neg': 0.8138791353098439, 'loss/num_matchable': 220.0266016404345, 'loss/num_unmatchable': 1090.2698958102417, 'loss/row_norm': 0.4761956669332394}
!!!!!sample_new_items: num_per_scene: 300
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
Not using the overlap matrices
[07/22/2024 10:02:58 gluefactory INFO] [E 3 | it 0] loss {total 3.308E+00, last 2.611E+00, assignment_nll 2.611E+00, nll_pos 4.673E+00, nll_neg 5.490E-01, num_matchable 5.820E+01, num_unmatchable 9.837E+02, confidence 7.844E-02, row_norm 4.042E-01}
[07/22/2024 10:13:00 gluefactory INFO] [E 3 | it 100] loss {total 3.242E+00, last 2.484E+00, assignment_nll 2.484E+00, nll_pos 4.416E+00, nll_neg 5.520E-01, num_matchable 1.060E+02, num_unmatchable 9.018E+02, confidence 1.402E-01, row_norm 3.995E-01}
[07/22/2024 10:22:52 gluefactory INFO] Intermediate training DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_intermediate_3_200_20240722_102252.pkl
[07/22/2024 10:22:52 gluefactory INFO] [E 3 | it 200] loss {total 3.342E+00, last 2.531E+00, assignment_nll 2.531E+00, nll_pos 4.480E+00, nll_neg 5.832E-01, num_matchable 6.022E+01, num_unmatchable 9.632E+02, confidence 1.089E-01, row_norm 4.015E-01}
[07/22/2024 10:33:31 gluefactory INFO] [E 3 | it 300] loss {total 3.193E+00, last 2.387E+00, assignment_nll 2.387E+00, nll_pos 4.242E+00, nll_neg 5.329E-01, num_matchable 7.836E+01, num_unmatchable 9.517E+02, confidence 1.267E-01, row_norm 4.194E-01}
[07/22/2024 10:43:18 gluefactory INFO] Intermediate training DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_intermediate_3_400_20240722_104318.pkl
[07/22/2024 10:43:18 gluefactory INFO] [E 3 | it 400] loss {total 3.745E+00, last 2.841E+00, assignment_nll 2.841E+00, nll_pos 5.144E+00, nll_neg 5.379E-01, num_matchable 8.641E+01, num_unmatchable 9.367E+02, confidence 1.365E-01, row_norm 4.209E-01}
[07/22/2024 10:43:57 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_3_55000.tar
[07/22/2024 10:52:55 gluefactory INFO] [E 3 | it 500] loss {total 5.881E-01, last 4.253E-01, assignment_nll 4.253E-01, nll_pos 7.682E-01, nll_neg 8.226E-02, num_matchable 6.438E+00, num_unmatchable 1.773E+02, confidence 1.225E-02, row_norm 7.365E-02}
debugging in data_to_log, iteraiton: 0
Data to be logged to training DataFrame: {'epoch': 3, 'iteration': 0, 'total': 0.623109757900238, 'last': 0.5035801529884338, 'assignment_nll': 0.5035801529884338, 'nll_pos': 0.9195927381515503, 'nll_neg': 0.08756757527589798, 'num_matchable': 7.40625, 'num_unmatchable': 163.578125, 'confidence': 0.011652616783976555, 'row_norm': 0.06988799571990967}
debugging in data_to_log, iteraiton: 100
Data to be logged to training DataFrame: {'epoch': 3, 'iteration': 100, 'total': 0.5212499499320984, 'last': 0.37160301208496094, 'assignment_nll': 0.37160301208496094, 'nll_pos': 0.6611356735229492, 'nll_neg': 0.08207030594348907, 'num_matchable': 27.71875, 'num_unmatchable': 134.6328125, 'confidence': 0.032412491738796234, 'row_norm': 0.06664881110191345}
debugging in data_to_log, iteraiton: 200
Data to be logged to training DataFrame: {'epoch': 3, 'iteration': 200, 'total': 0.548249363899231, 'last': 0.4176313579082489, 'assignment_nll': 0.4176313579082489, 'nll_pos': 0.7444356083869934, 'nll_neg': 0.0908270925283432, 'num_matchable': 9.828125, 'num_unmatchable': 167.4453125, 'confidence': 0.015637848526239395, 'row_norm': 0.0716637596487999}
debugging in data_to_log, iteraiton: 300
Data to be logged to training DataFrame: {'epoch': 3, 'iteration': 300, 'total': 0.5781821012496948, 'last': 0.4436582326889038, 'assinll': 1.9955357313156128, 'nll_pos': 3.5395402908325195, 'nll_neg': 0.451531320810318, 'num_matchable': 54.796875, 'num_unmatchable': 812.9921875, 'confidence': 0.1018897145986557, 'row_norm': 0.34669986367225647}
debugging in data_to_log, iteraiton: 400
Data to be logged to training DataFrame: {'epoch': 3, 'iteration': 400, 'total': 3.1240272521972656, 'last': 2.3726673126220703, 'assignment_nll': 2.3726673126220703, 'nll_pos': 4.298075199127197, 'nll_neg': 0.44725948572158813, 'num_matchable': 76.84375, 'num_unmatchable': 767.109375, 'confidence': 0.11822553724050522, 'row_norm': 0.3490593731403351}
debugging in data_to_log, iteraiton: 500
Data to be logged to training DataFrame: {'epoch': 3, 'iteration': 500, 'total': 2.6392905712127686, 'last': 1.926247477531433, 'assignment_nll': 1.926247477531433, 'nll_pos': 3.4230451583862305, 'nll_neg': 0.4294494390487671, 'num_matchable': 52.359375, 'num_unmatchable': 825.7109375, 'confidence': 0.08253830671310425, 'row_norm': 0.35839951038360596}
[07/22/2024 11:06:24 gluefactory INFO] Intermediate DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_20240722_110624.pkl
[07/22/2024 11:06:24 gluefactory INFO] [Validation] {match_recall 1.301E-01, match_precision 2.312E-01, accuracy 8.225E-01, average_precision 5.313E-02, loss/total 3.498E+00, loss/last 3.498E+00, loss/assignment_nll 3.498E+00, loss/nll_pos 6.258E+00, loss/nll_neg 7.386E-01, loss/num_matchable 2.200E+02, loss/num_unmatchable 1.090E+03, loss/row_norm 5.160E-01}
[07/22/2024 11:31:09 gluefactory INFO] Intermediate training DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_intermediate_3_600_20240722_113109.pkl
[07/22/2024 11:31:09 gluefactory INFO] [E 3 | it 600] loss {total 2.734E+00, last 2.038E+00, assignment_nll 2.038E+00, nll_pos 3.581E+00, nll_neg 4.959E-01, num_matchable 6.869E+01, num_unmatchable 8.022E+02, confidence 9.561E-02, row_norm 3.232E-01}
[07/22/2024 11:44:15 gluefactory INFO] [E 3 | it 700] loss {total 2.655E+00, last 1.974E+00, assignment_nll 1.974E+00, nll_pos 3.529E+00, nll_neg 4.187E-01, num_matchable 5.511E+01, num_unmatchable 8.336E+02, confidence 8.227E-02, row_norm 3.583E-01}
[07/22/2024 11:56:13 gluefactory INFO] Intermediate training DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_intermediate_3_800_20240722_115613.pkl
[07/22/2024 11:56:13 gluefactory INFO] [E 3 | it 800] loss {total 2.761E+00, last 2.110E+00, assignment_nll 2.110E+00, nll_pos 3.820E+00, nll_neg 3.992E-01, num_matchable 1.089E+02, num_unmatchable 7.560E+02, confidence 1.145E-01, row_norm 3.594E-01}
[07/22/2024 12:06:44 gluefactory INFO] [E 3 | it 900] loss {total 2.676E+00, last 1.962E+00, assignment_nll 1.962E+00, nll_pos 3.476E+00, nll_neg 4.487E-01, num_matchable 6.422E+01, num_unmatchable 8.033E+02, confidence 1.194E-01, row_norm 3.538E-01}
[07/22/2024 12:19:06 gluefactory INFO] Intermediate training DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_intermediate_3_1000_20240722_121906.pkl
[07/22/2024 12:19:06 gluefactory INFO] [E 3 | it 1000] loss {total 1.814E+00, last 1.388E+00, assignment_nll 1.388E+00, nll_pos 2.511E+00, nll_neg 2.656E-01, num_matchable 3.164E+01, num_unmatchable 4.892E+02, confidence 6.487E-02, row_norm 2.147E-01}
Data to be logged: {'epoch': 3, 'iteration': 500, 'match_recall': 0.13011813131436653, 'match_precision': 0.23122422915010674, 'accuracy': 0.8224832376866139, 'average_precision': 0.05313483808128391, 'loss/total': 3.498316009830567, 'loss/last': 3.498316009830567, 'loss/assignment_nll': 3.498316009830567, 'loss/nll_pos': 6.258002335944373, 'loss/nll_neg': 0.7386296695522341, 'loss/num_matchable': 220.0266016404345, 'loss/num_unmatchable': 1090.2698958102417, 'loss/row_norm': 0.5160016513034632}
debugging in data_to_log, iteraiton: 600
Data to be logged to training DataFrame: {'epoch': 3, 'iteration': 600, 'total': 2.7337193489074707, 'last': 2.0382301807403564, 'assignment_nll': 2.0382301807403564, 'nll_pos': 3.5805249214172363, 'nll_neg': 0.49593567848205566, 'num_matchable': 68.6875, 'num_unmatchable': 802.25, 'confidence': 0.09560719132423401, 'row_norm': 0.3232366144657135}
debugging in data_to_log, iteraiton: 700
Data to be logged to training DataFrame: {'epoch': 3, 'iteration': 700, 'total': 2.654604196548462, 'last': 1.9738798141479492, 'assignment_nll': 1.9738798141479492, 'nll_pos': 3.529048442840576, 'nll_neg': 0.4187116324901581, 'num_matchable': 55.109375, 'num_unmatchable': 833.6171875, 'confidence': 0.08227404206991196, 'row_norm': 0.3582572340965271}
debugging in data_to_log, iteraiton: 800
Data to be logged to training DataFrame: {'epoch': 3, 'iteration': 800, 'total': 2.7608985900878906, 'last': 2.1097497940063477, 'assignment_nll': 2.1097497940063477, 'nll_pos': 3.820338487625122, 'nll_neg': 0.39916104078292847, 'num_matchable': 108.90625, 'num_unmatchable': 755.96875, 'confidence': 0.1145496666431427, 'row_norm': 0.3594473600387573}
debugging in data_to_log, iteraiton: 900
Data to be logged to training DataFrame: {'epoch': 3, 'iteration': 900, 'total': 2.675753593444824, 'last': 1.962198257446289, 'assignment_nll': 1.962198257446289, 'nll_pos': 3.4756672382354736, 'nll_neg': 0.44872939586639404, 'num_matchable': 64.21875, 'num_unmatchable': 803.2578125, 'confidence': 0.11937056481838226, 'row_norm': 0.3537551462650299}
debugging in data_to_log, iteraiton: 1000
Data to be logged to training DataFrame: {'epoch': 3, 'iteration': 1000, 'total': 2.9059524536132812, 'last': 2.2222394943237305, 'assignment_nll': 2.2222394943237305, 'nll_pos': 3.979397773742676, 'nll_neg': 0.46508097648620605, 'num_matchable': 55.75, 'num_unmatchable': 809.84375, 'confidence': 0.10995938628911972, 'row_norm': 0.3463800251483917}
[07/22/2024 12:32:07 gluefactory INFO] Intermediate DataFrame saved to: /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_21-07-treedepth-grayscale-64-colmapPose-cache/pretrain_lightglue_20240722_123207.pkl
[07/22/2024 12:32:07 gluefactory INFO] [Validation] {match_recall 1.351E-01, match_precision 2.360E-01, accuracy 8.229E-01, average_precision 5.612E-02, loss/total 3.488E+00, loss/last 3.488E+00, loss/assignment_nll 3.488E+00, loss/nll_pos 6.197E+00, loss/nll_neg 7.791E-01, loss/num_matchable 2.200E+02, loss/num_unmatchable 1.090E+03, loss/row_norm 4.995E-01}
[07/22/2024 12:46:21 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_3_60000.tar
Data to be logged: {'epoch': 3, 'iteration': 1000, 'match_recall': 0.13509755938228635, 'match_precision': 0.23603172415862905, 'accuracy': 0.8229307315482239, 'average_precision': 0.05611886172879007, 'loss/total': 3.487856042858133, 'loss/last': 3.487856042858133, 'loss/assignment_nll': 3.487856042858133, 'loss/nll_pos': 6.196623158861861, 'loss/nll_neg': 0.7790889509023887, 'loss/num_matchable': 220.0266016404345, 'loss/num_unmatchable': 1090.2698958102417, 'loss/row_norm': 0.4994556612092147}
[07/22/2024 12:46:33 gluefactory ERROR] CUDA Out of Memory Error ???
[07/22/2024 12:46:33 gluefactory ERROR] Hostname: gpu02.doc.ic.ac.uk
[07/22/2024 12:46:33 gluefactory ERROR] Error occurred at: 2024-07-22 12:46:33
[07/22/2024 12:46:33 gluefactory ERROR] Unable to get GPU Memory Usage: 'allocated_bytes.all.current'
Traceback (most recent call last):
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/GlueFactory/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/GlueFactory/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/train.py", line 1192, in <module>
    torch.multiprocessing.spawn(
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/GlueFactory/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 281, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/GlueFactory/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 237, in start_processes
    while not context.join():
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/GlueFactory/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 188, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException: 

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/GlueFactory/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 75, in _wrap
    fn(i, *args)
  File "/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/train.py", line 1105, in main_worker
    training(rank, conf, output_dir, args, results_df, train_df)
  File "/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/train.py", line 748, in training
    pred = model(data)
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/GlueFactory/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/GlueFactory/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/GlueFactory/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1589, in forward
    inputs, kwargs = self._pre_forward(*inputs, **kwargs)
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/GlueFactory/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1487, in _pre_forward
    self._sync_buffers()
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/GlueFactory/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 2129, in _sync_buffers
    self._sync_module_buffers(authoritative_rank)
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/GlueFactory/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 2133, in _sync_module_buffers
    self._default_broadcast_coalesced(authoritative_rank=authoritative_rank)
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/GlueFactory/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 2155, in _default_broadcast_coalesced
    self._distributed_broadcast_coalesced(bufs, bucket_size, authoritative_rank)
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/GlueFactory/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 2070, in _distributed_broadcast_coalesced
    dist._broadcast_coalesced(
RuntimeError: [../third_party/gloo/gloo/transport/tcp/pair.cc:534] Connection closed by peer [146.169.53.211]:29337

./third_party/gloo/gloo/transport/tcp/pair.cc:534] Connection closed by peer [146.169.52.73]:10890