Gotchas

When we think about how to use a set of criteria like those described here, it is very easy to focus on certain kinds of systems. For example, if we go straight to “How to make a linter” here, we run the risk of limiting the evaluation to a kind of low-hanging fruit. Code is created in a variety of environments yet we run the risk of perhaps focusing on the conventions used commonly on Github, for example, and introducing bias into the evaluation products.

So let’s try not to optimize prematurely here.

Finally, we are interested here in the outcomes of these evaluation efforts and in the code environments found in research groups, whether in university, federal or other independent organizations closely tied to those spaces.