Reproduction usually stalls not because the method is difficult but because key information is missing from the paper. Here are the six most common gaps, how to judge whether a paper is worth attempting, and what to do when the numbers will not match.
Reproducing a paper usually goes like this. The method section reads clearly, you implement it faithfully, and the numbers do not match. You check your implementation repeatedly and find nothing wrong. Most of the time nothing is wrong with it. A paper is written so the method can be understood, not so the experiment can be rerun. Journals impose length limits, authors assume domain knowledge, and information that is essential for reproduction gets compressed out. The six most common gaps 1. Hyperparameters given only in part. Learning rate and batch size appear; the optimizer configuration, the schedule, and how the epoch count was chosen do not. Those were defaults to the original authors, and defaults differ between people. 2. Preprocessing described in one clause. "The data were normalized." Along which axis? Before or after the train-test split? The latter leaks information, and the two choices can differ by several percentage points. 3. Unspecified data splits. How train and test were divided, what seed was used, whether the split was stratified on some variable. With small samples, a different seed can flip the conclusion. 4. Ambiguous metric definitions. One metric name can mean several computations. Is a multi-class F1 macro-averaged or micro-averaged? That gap is often larger than the improvement the method claims. 5. Which run is being reported. The mean across runs, or the best one? Papers often state that five runs were performed without saying whether the table shows the average or the maximum. 6. Unwritten engineering. Certain samples dropped during cleaning, a trick used during training, a restart procedure for non-convergence. Trivia to the author, decisive for anyone reproducing. Judging whether a paper is worth the attempt Assess before committing, so you do not spend days discovering it was impossible: Is the data obtainable? A public dataset is ideal. Proprietary data with described statistical properties can be simulated. With neither, you can validate implementation logic but cannot reconcile numbers. How complete is the method description? Split it into data handling, model, training, and evaluation, and check whether each becomes an executable step list. If two or more blocks cannot, difficulty rises sharply. Is there released code? Then it is not a reproduction problem but an environment problem, which is a different exercise entirely. Are the results unusually strong? Improvements far beyond contemporaneous work either reflect something genuinely novel or something unstated. Worth reproducing, but lower your expectations. When the numbers will not match This is where attempts go wrong. The common mistake is tuning parameters until the number approaches the published one . That produces nothing of value: you have found a configuration that reaches a figure, not reproduced a method. Treat the discrepancy as information instead: 1. Confirm your implementation is correct in a simple case , using a small example with a known answer 2. Isolate block by block : do the intermediate results after preprocessing match the description? Does your model parameter count match the reported one? 3. Record and explain the divergence : what reproduced, what did not, which avenues you checked, and the most likely cause A failed reproduction is itself a valid finding , provided you can say which step it failed at. The field-wide push on reproducibility over recent years was built largely from accumulated records of exactly this kind. Where a tool helps The repetitive parts are: read the method, decompose it into steps, enumerate what is missing, write the code, run it, reconcile, and localize the differences. Those can be delegated, and that is the span our paper reproduction agent covers. Two things must stay with a person: which reading to adopt where the paper is unclear , which takes domain judgement, and how close counts as reproduced , which is a question of scholarly standards. So the workable division is: the tool enumerates each gap with a proposed value and you decide, then the tool produces an item-by-item reconciliation and you judge whether it constitutes reproduction. The important part is not letting the tool paper over the difference. A tool that quietly tunes until the numbers agree is more dangerous than one that fails to reproduce. Related : Empirical data analysis covers making the analysis process itself checkable.