Some numbers from a language model are computed. Others are generated because they look about right. The two are nearly indistinguishable on the page and completely different in a report. Here is how to tell them apart and how to make an analysis checkable.
Hand a spreadsheet to a language model, ask whether two groups differ significantly, and you will get a p-value. The question worth asking is: was that p-value computed, or written? This is not pedantry. A language model predicts the next token. When it has not actually executed anything, the number it produces reflects what such a figure usually looks like, not what this dataset yields. The two are visually identical, both a decimal with a confident sentence attached. The only difference is that one can be reproduced and the other cannot. 1. Quick tells A few signals separate them fast. Is there code? If a calculation genuinely ran, code exists. A conclusion with no code behind it is almost certainly generated. Suspiciously tidy precision. Generated figures tend to be too clean: p = 0.03, correlation 0.75, a 15% improvement. Real output tends to look like 0.0287 or 0.7413. The reverse does not hold, though, since a generator can produce untidy numbers too. Change the data and see if the answer moves. This is the hardest test. Alter a few rows substantially and ask again. If the result does not budge, nothing read your data. Does it mention anything specific about your data? Something that actually read the file tends to notice a missing value in row 47 or two clear outliers in a column. An analysis that says nothing about the state of the data probably never looked at it. 2. What a checkable analysis contains An analysis you can rely on has four parts: 1. A data health check : row count, missingness, outliers, and the size of each group 2. A justified method choice : why this test, what it assumes, and whether your data meets those assumptions 3. Runnable code : a complete script, not fragments 4. Raw output : what the program printed, not a paraphrase of it With those four you can rerun it yourself. Rerunnable is what checkable means. 3. The real risk is silent handling, not arithmetic errors Outright miscalculation is the lesser danger, because wrong answers usually look wrong. The dangerous cases are quiet: Missing values dropped silently. Two hundred of a thousand rows lack a variable, those rows disappear, and the remaining eight hundred no longer represent the original population. Unless you are told, you will never know. Outliers silently kept or silently removed. Keeping them drags the mean; removing them flatters the result. Either can be defensible, but it has to be stated. No correction for multiple comparisons. Compare twenty groups and one will be significant. Without knowing how many comparisons were run, that significance means nothing. Grouping criteria drifting mid-job. On large batches, an inconsistent standard makes the final numbers incomparable. Judging an analysis by how clean its conclusion looks is far weaker than checking whether it volunteered these details. A good report contains a passage on what was done to the data, why, and what that might do to the result. 4. Ask up front In practice, stating requirements at the start beats checking afterwards. These instructions work well: Before analyzing, give me the data health check: row count, missingness, outliers. Compute with code, and give me the code and its raw output. Tell me explicitly how you handled missing values and outliers. Do not apply defaults silently. Separate what the evidence supports from what is only a trend. That last one matters most. Many flawed analyses have sound numbers and simply present a trend as a conclusion. 5. What the tool should be responsible for Those principles are how we built empirical data analysis: analysis is performed by writing and actually executing code, with the code and intermediate output delivered; the data health check runs before modelling; and when a script errors or produces implausible output the agent goes back and fixes it rather than submitting the broken result. This is not rigour for its own sake. An analysis that cannot be re-verified has no use in research or decision making. A number you cannot check is barely better than no number at all. Related : Paper reproduction covers actually running a published method and reconciling it against the reported results.