OpenAI’s policies hinder reproducible research on language models

Researchers rely on ML models created by companies to conduct research. Codex, like other OpenAI models, is not open source, so users rely on OpenAI for accessing the model. Scientific research already suffers from a reproducibility crisis, including in fields that use ML. Since small changes in a model can result in significant downstream effects, a prerequisite for reproducible research is access to the exact model used in an experiment. OpenAI regularly updates newer models, such as GPT-3.5 and GPT-4, so the use of those models is automatically a barrier to reproducibility. That means the prospects for reproducible research using the newer models are also dim-to-nonexistent. OpenAI isn't responsibly maintaining this infrastructure by providing versioned models. OpenAI asked researchers to switch to GPT 3.5 models.