Skip to content

/ CodeSearchNet

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Shuffle Validation Set when calculating MRR in included scripts #87

Closed
hamelsmu opened this issue Nov 18, 2019 · 2 comments
Closed

Shuffle Validation Set when calculating MRR in included scripts #87

hamelsmu opened this issue Nov 18, 2019 · 2 comments

Comments

@hamelsmu
Copy link
Member

@hamelsmu hamelsmu commented Nov 18, 2019

from @mallamanis

One masters' student in Berkeley has asked me the following question for CodeSearchNet.

The validation loss logging shows that the MRR performance decreases as it is being computed (at the end of the epoch). This seems to be the case with many of the runs on W&B. Do you have any idea why this might happening? I don't see anything obviously wrong.

For example,

11 (valid): Batch     0 (has 200 samples). Processed 0 samples. Loss so far: 0.0000.  MRR so far: 0.0000 
11 (valid): Batch     1 (has 200 samples). Processed 200 samples. Loss so far: 2.3159.  MRR so far: 0.5468 
11 (valid): Batch     2 (has 200 samples). Processed 400 samples. Loss so far: 2.3237.  MRR so far: 0.5623 
11 (valid): Batch     3 (has 200 samples). Processed 600 samples. Loss so far: 2.3163.  MRR so far: 0.5652 
11 (valid): Batch     4 (has 200 samples). Processed 800 samples. Loss so far: 2.3615.  MRR so far: 0.5568 
11 (valid): Batch     5 (has 200 samples). Processed 1000 samples. Loss so far: 2.5153.  MRR so far: 0.5323 
11 (valid): Batch     6 (has 200 samples). Processed 1200 samples. Loss so far: 2.8651.  MRR so far: 0.4921 
11 (valid): Batch     7 (has 200 samples). Processed 1400 samples. Loss so far: 2.7642.  MRR so far: 0.5085 
11 (valid): Batch     8 (has 200 samples). Processed 1600 samples. Loss so far: 2.7468.  MRR so far: 0.5091 
11 (valid): Batch     9 (has 200 samples). Processed 1800 samples. Loss so far: 2.7153.  MRR so far: 0.5134 
11 (valid): Batch    10 (has 200 samples). Processed 2000 samples. Loss so far: 2.7024.  MRR so far: 0.5131 
11 (valid): Batch    11 (has 200 samples). Processed 2200 samples. Loss so far: 2.7061.  MRR so far: 0.5125 
11 (valid): Batch    12 (has 200 samples). Processed 2400 samples. Loss so far: 2.6665.  MRR so far: 0.5183 
11 (valid): Batch    13 (has 200 samples). Processed 2600 samples. Loss so far: 2.6986.  MRR so far: 0.5157 
11 (valid): Batch    14 (has 200 samples). Processed 2800 samples. Loss so far: 2.6975.  MRR so far: 0.5166 
11 (valid): Batch    15 (has 200 samples). Processed 3000 samples. Loss so far: 2.7363.  MRR so far: 0.5118 
11 (valid): Batch    16 (has 200 samples). Processed 3200 samples. Loss so far: 2.7226.  MRR so far: 0.5137 
11 (valid): Batch    17 (has 200 samples). Processed 3400 samples. Loss so far: 2.7146.  MRR so far: 0.5153 
11 (valid): Batch    18 (has 200 samples). Processed 3600 samples. Loss so far: 2.7491.  MRR so far: 0.5115 
11 (valid): Batch    19 (has 200 samples). Processed 3800 samples. Loss so far: 2.7468.  MRR so far: 0.5108 
11 (valid): Batch    20 (has 200 samples). Processed 4000 samples. Loss so far: 2.7470.  MRR so far: 0.5097 
11 (valid): Batch    21 (has 200 samples). Processed 4200 samples. Loss so far: 2.7783.  MRR so far: 0.5070 
11 (valid): Batch    22 (has 200 samples). Processed 4400 samples. Loss so far: 2.7725.  MRR so far: 0.5086 
11 (valid): Batch    23 (has 200 samples). Processed 4600 samples. Loss so far: 2.7606.  MRR so far: 0.5096 
11 (valid): Batch    24 (has 200 samples). Processed 4800 samples. Loss so far: 2.7733.  MRR so far: 0.5069 
11 (valid): Batch    25 (has 200 samples). Processed 5000 samples. Loss so far: 2.8067.  MRR so far: 0.5030 
11 (valid): Batch    26 (has 200 samples). Processed 5200 samples. Loss so far: 2.7878.  MRR so far: 0.5054 
11 (valid): Batch    27 (has 200 samples). Processed 5400 samples. Loss so far: 2.7869.  MRR so far: 0.5054 
11 (valid): Batch    28 (has 200 samples). Processed 5600 samples. Loss so far: 2.8128.  MRR so far: 0.5003 
11 (valid): Batch    29 (has 200 samples). Processed 5800 samples. Loss so far: 2.8420.  MRR so far: 0.4959 
11 (valid): Batch    30 (has 200 samples). Processed 6000 samples. Loss so far: 2.8311.  MRR so far: 0.4981 
11 (valid): Batch    31 (has 200 samples). Processed 6200 samples. Loss so far: 2.8291.  MRR so far: 0.4978 
11 (valid): Batch    32 (has 200 samples). Processed 6400 samples. Loss so far: 2.8190.  MRR so far: 0.4993 
11 (valid): Batch    33 (has 200 samples). Processed 6600 samples. Loss so far: 2.8345.  MRR so far: 0.4980 
11 (valid): Batch    34 (has 200 samples). Processed 6800 samples. Loss so far: 2.8100.  MRR so far: 0.5011 
11 (valid): Batch    35 (has 200 samples). Processed 7000 samples. Loss so far: 2.7998.  MRR so far: 0.5026 
11 (valid): Batch    36 (has 200 samples). Processed 7200 samples. Loss so far: 2.7841.  MRR so far: 0.5052 
11 (valid): Batch    37 (has 200 samples). Processed 7400 samples. Loss so far: 2.7836.  MRR so far: 0.5052 
11 (valid): Batch    38 (has 200 samples). Processed 7600 samples. Loss so far: 2.7906.  MRR so far: 0.5044 
11 (valid): Batch    39 (has 200 samples). Processed 7800 samples. Loss so far: 2.8042.  MRR so far: 0.5020 
11 (valid): Batch    40 (has 200 samples). Processed 8000 samples. Loss so far: 2.8095.  MRR so far: 0.5013 
11 (valid): Batch    41 (has 200 samples). Processed 8200 samples. Loss so far: 2.8119.  MRR so far: 0.5008 
11 (valid): Batch    42 (has 200 samples). Processed 8400 samples. Loss so far: 2.7931.  MRR so far: 0.5038 
11 (valid): Batch    43 (has 200 samples). Processed 8600 samples. Loss so far: 2.7902.  MRR so far: 0.5043 
11 (valid): Batch    44 (has 200 samples). Processed 8800 samples. Loss so far: 2.7918.  MRR so far: 0.5041 
11 (valid): Batch    45 (has 200 samples). Processed 9000 samples. Loss so far: 2.7948.  MRR so far: 0.5044 
11 (valid): Batch    46 (has 200 samples). Processed 9200 samples. Loss so far: 2.7991.  MRR so far: 0.5031 
11 (valid): Batch    47 (has 200 samples). Processed 9400 samples. Loss so far: 2.8023.  MRR so far: 0.5028 
11 (valid): Batch    48 (has 200 samples). Processed 9600 samples. Loss so far: 2.8052.  MRR so far: 0.5020 
11 (valid): Batch    49 (has 200 samples). Processed 9800 samples. Loss so far: 2.8261.  MRR so far: 0.4986 
11 (valid): Batch    50 (has 200 samples). Processed 10000 samples. Loss so far: 2.8538.  MRR so far: 0.4944 
11 (valid): Batch    51 (has 200 samples). Processed 10200 samples. Loss so far: 2.8511.  MRR so far: 0.4946 
11 (valid): Batch    52 (has 200 samples). Processed 10400 samples. Loss so far: 2.8531.  MRR so far: 0.4949 
11 (valid): Batch    53 (has 200 samples). Processed 10600 samples. Loss so far: 2.8617.  MRR so far: 0.4928 
11 (valid): Batch    54 (has 200 samples). Processed 10800 samples. Loss so far: 2.8791.  MRR so far: 0.4897 
11 (valid): Batch    55 (has 200 samples). Processed 11000 samples. Loss so far: 2.8967.  MRR so far: 0.4876 
11 (valid): Batch    56 (has 200 samples). Processed 11200 samples. Loss so far: 2.9021.  MRR so far: 0.4871 
11 (valid): Batch    57 (has 200 samples). Processed 11400 samples. Loss so far: 2.9030.  MRR so far: 0.4873 
11 (valid): Batch    58 (has 200 samples). Processed 11600 samples. Loss so far: 2.9257.  MRR so far: 0.4843 
11 (valid): Batch    59 (has 200 samples). Processed 11800 samples. Loss so far: 2.9270.  MRR so far: 0.4841 
11 (valid): Batch    60 (has 200 samples). Processed 12000 samples. Loss so far: 2.9342.  MRR so far: 0.4836 
11 (valid): Batch    61 (has 200 samples). Processed 12200 samples. Loss so far: 2.9443.  MRR so far: 0.4819 
11 (valid): Batch    62 (has 200 samples). Processed 12400 samples. Loss so far: 2.9565.  MRR so far: 0.4798 
11 (valid): Batch    63 (has 200 samples). Processed 12600 samples. Loss so far: 2.9494.  MRR so far: 0.4810 
11 (valid): Batch    64 (has 200 samples). Processed 12800 samples. Loss so far: 2.9631.  MRR so far: 0.4785 
11 (valid): Batch    65 (has 200 samples). Processed 13000 samples. Loss so far: 2.9674.  MRR so far: 0.4779 
11 (valid): Batch    66 (has 200 samples). Processed 13200 samples. Loss so far: 2.9685.  MRR so far: 0.4778 
11 (valid): Batch    67 (has 200 samples). Processed 13400 samples. Loss so far: 2.9789.  MRR so far: 0.4769 
11 (valid): Batch    68 (has 200 samples). Processed 13600 samples. Loss so far: 2.9794.  MRR so far: 0.4765 
11 (valid): Batch    69 (has 200 samples). Processed 13800 samples. Loss so far: 2.9754.  MRR so far: 0.4766 
11 (valid): Batch    70 (has 200 samples). Processed 14000 samples. Loss so far: 2.9731.  MRR so far: 0.4767 
11 (valid): Batch    71 (has 200 samples). Processed 14200 samples. Loss so far: 2.9822.  MRR so far: 0.4751 
11 (valid): Batch    72 (has 200 samples). Processed 14400 samples. Loss so far: 2.9755.  MRR so far: 0.4756 
11 (valid): Batch    73 (has 200 samples). Processed 14600 samples. Loss so far: 2.9747.  MRR so far: 0.4758 
11 (valid): Batch    74 (has 200 samples). Processed 14800 samples. Loss so far: 2.9666.  MRR so far: 0.4770 
11 (valid): Batch    75 (has 200 samples). Processed 15000 samples. Loss so far: 2.9757.  MRR so far: 0.4758 
11 (valid): Batch    76 (has 200 samples). Processed 15200 samples. Loss so far: 2.9775.  MRR so far: 0.4756 
11 (valid): Batch    77 (has 200 samples). Processed 15400 samples. Loss so far: 2.9813.  MRR so far: 0.4745 
11 (valid): Batch    78 (has 200 samples). Processed 15600 samples. Loss so far: 2.9830.  MRR so far: 0.4739 
11 (valid): Batch    79 (has 200 samples). Processed 15800 samples. Loss so far: 2.9905.  MRR so far: 0.4726 
11 (valid): Batch    80 (has 200 samples). Processed 16000 samples. Loss so far: 3.0173.  MRR so far: 0.4688 
11 (valid): Batch    81 (has 200 samples). Processed 16200 samples. Loss so far: 3.0520.  MRR so far: 0.4645 
11 (valid): Batch    82 (has 200 samples). Processed 16400 samples. Loss so far: 3.0599.  MRR so far: 0.4630 
11 (valid): Batch    83 (has 200 samples). Processed 16600 samples. Loss so far: 3.0639.  MRR so far: 0.4625 
11 (valid): Batch    84 (has 200 samples). Processed 16800 samples. Loss so far: 3.0691.  MRR so far: 0.4616 
11 (valid): Batch    85 (has 200 samples). Processed 17000 samples. Loss so far: 3.0723.  MRR so far: 0.4614 
11 (valid): Batch    86 (has 200 samples). Processed 17200 samples. Loss so far: 3.0691.  MRR so far: 0.4615 
11 (valid): Batch    87 (has 200 samples). Processed 17400 samples. Loss so far: 3.0919.  MRR so far: 0.4589 
11 (valid): Batch    88 (has 200 samples). Processed 17600 samples. Loss so far: 3.0917.  MRR so far: 0.4587 
11 (valid): Batch    89 (has 200 samples). Processed 17800 samples. Loss so far: 3.0887.  MRR so far: 0.4592 
11 (valid): Batch    90 (has 200 samples). Processed 18000 samples. Loss so far: 3.1092.  MRR so far: 0.4561 
11 (valid): Batch    91 (has 200 samples). Processed 18200 samples. Loss so far: 3.1391.  MRR so far: 0.4515 
11 (valid): Batch    92 (has 200 samples). Processed 18400 samples. Loss so far: 3.1650.  MRR so far: 0.4482 
11 (valid): Batch    93 (has 200 samples). Processed 18600 samples. Loss so far: 3.1630.  MRR so far: 0.4486 
11 (valid): Batch    94 (has 200 samples). Processed 18800 samples. Loss so far: 3.1624.  MRR so far: 0.4488 
11 (valid): Batch    95 (has 200 samples). Processed 19000 samples. Loss so far: 3.1692.  MRR so far: 0.4480 
11 (valid): Batch    96 (has 200 samples). Processed 19200 samples. Loss so far: 3.1660.  MRR so far: 0.4486 
11 (valid): Batch    97 (has 200 samples). Processed 19400 samples. Loss so far: 3.1729.  MRR so far: 0.4478 
11 (valid): Batch    98 (has 200 samples). Processed 19600 samples. Loss so far: 3.1779.  MRR so far: 0.4468 
11 (valid): Batch    99 (has 200 samples). Processed 19800 samples. Loss so far: 3.1967.  MRR so far: 0.4445 
11 (valid): Batch   100 (has 200 samples). Processed 20000 samples. Loss so far: 3.1978.  MRR so far: 0.4443 
11 (valid): Batch   101 (has 200 samples). Processed 20200 samples. Loss so far: 3.1949.  MRR so far: 0.4443 
11 (valid): Batch   102 (has 200 samples). Processed 20400 samples. Loss so far: 3.1932.  MRR so far: 0.4444 
11 (valid): Batch   103 (has 200 samples). Processed 20600 samples. Loss so far: 3.1864.  MRR so far: 0.4453 
11 (valid): Batch   104 (has 200 samples). Processed 20800 samples. Loss so far: 3.1881.  MRR so far: 0.4453 
11 (valid): Batch   105 (has 200 samples). Processed 21000 samples. Loss so far: 3.1855.  MRR so far: 0.4457 
11 (valid): Batch   106 (has 200 samples). Processed 21200 samples. Loss so far: 3.1818.  MRR so far: 0.4460 
11 (valid): Batch   107 (has 200 samples). Processed 21400 samples. Loss so far: 3.1841.  MRR so far: 0.4458 
11 (valid): Batch   108 (has 200 samples). Processed 21600 samples. Loss so far: 3.1811.  MRR so far: 0.4462 
11 (valid): Batch   109 (has 200 samples). Processed 21800 samples. Loss so far: 3.1806.  MRR so far: 0.4463 
11 (valid): Batch   110 (has 200 samples). Processed 22000 samples. Loss so far: 3.1816.  MRR so far: 0.4464 
11 (valid): Batch   111 (has 200 samples). Processed 22200 samples. Loss so far: 3.1796.  MRR so far: 0.4467 
11 (valid): Batch   112 (has 200 samples). Processed 22400 samples. Loss so far: 3.1842.  MRR so far: 0.4457 
  Epoch 11 (valid) took 6.12s [processed 3689 samples/second]
 Validation:  Loss: 3.190802 | MRR: 0.444851
@hamelsmu hamelsmu changed the title Check MRR calculation in included scripts Shuffle Validation Set when calculating MRR in included scripts Nov 19, 2019
@fengzhangyin
Copy link
Contributor

@fengzhangyin fengzhangyin commented Dec 10, 2019

Hi @hamelsmu @mallamanis ,

According to my understanding, the order of the data should be fixed when we calculate MRR.
@mallamanis mentioned this here.

But I find that this is not the case in the code. In model_test.py at line 82, the data is shuffled.
May be this place caused the error?
@mallamanis
Copy link
Collaborator

@mallamanis mallamanis commented Dec 17, 2019

Hi @fengzhangyin,

I wouldn't expect this to be the case. Yes, the data is shuffled here but the numpy seed is fixed to 0 (here) which should yield a fixed "random" ordering.

I still haven't found a reason that the loss seems to be monotonically increasing... It's worth investigating more, of course.
@hamelsmu hamelsmu closed this Sep 4, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
@mallamanis @hamelsmu @fengzhangyin
You can’t perform that action at this time.