Draft Software Assessment Guidelines
We ask that you consider a few things when looking over these guidelines.
- Is the statement evaluatable by someone external to the project? When reading the criterion, can you think of some feature common to code files, code repositories, documentation, or a project website you would look for? If not, do you consider the concept referenced in the statement to still be important to code or project quality?
- Does the criterion's concept reflect an ideal endpoint? Can you suggest meaningful and evaluatable criteria that are actionable community practices that lead to that ideal (and are those reflected in other criteria here)? What is the happy path?
- Is the criterion (or the group) related to active development efforts or to preservation/maintenance efforts? Consider a research project lifecycle.
- Is the criterion or group associated with a codebase or the project information? Does it relate to code maturity or project maturity and is that clear from the statement?
- For any criteria directly related to code, can the criterion be applied to any codebase or software artifact or does it apply only to one (or a few) types of software, such as a plugin/extension, module/library, web application, web service, desktop application, etc?
- Is the criterion *directly* related to the software/code developed for the project, eg is it the deliverable? The aim is to not assess the framework used, for example, but the plugin or customizations made explicitly for the project.
Finally, consider the criteria in relation to a project or codebase you're familiar with.
Table of Contents
We are making a distinction between a project developed within common open source models and a project that is open, meeting EarthCube's requirements, to meet publication, reproducibility and provenance concerns.
About the layout
The layout used in the draft is meant to provide the first iteration of feedback for comment.
Criteria Options
The original statement.
The original statement, flagged for review.
Suggested revisions to the original statement or a replacement for the original. The original statement is provided for reference.
Suggested new statement.
Suggested revision is to delete the statement entirely.
Criteria Descriptors
The descriptors provide additional context for the criterion if applicable.
Application: indicates whether a statement refers to only the codebase, the project as a whole (generally relates to project artifacts not part of the codebase such as a project website), or applies to both.
Grouping: describes a logical grouping to the criteria within a subcategory.
Notes: any notes provided by he initial reviewers regarding suggested revisions.
References: link to a citation providing a rationale for the criterion.
Similar Criteria: links to criteria listed in other subcategories that may be duplicated concepts.
Interoperability
Ed. note: criteria needs include project-level criteria (serving data services, consuming data services), validation, cross-project criteria (if any can be identified and evaluated).
Provides tests demonstrating compliance to open standards.
Uses mature, ratified, non-draft open standards.
Uses open standards.
[Placeholder - implements JSON-LD, etc.]
Licensing
Each source code file or document has a licence header; each web page has a rights and authorship footer.
License is specified in each source code file or document.
Project website includes authorship information.
Each code repository README (or similar) indicates license applied.
Each code repository README (or similar) includes authorship information.
Site or product content has a 'free for public use' license or statement.
Project website has a 'free for public use' license or statement.
Site or product content has a license.
Project website has a license.
Software (source and binaries) has a licence.
Software has an Open Software Initiative (OSI) recognised licence.
Software has an open source licence.
Web site states applicable licences.
Documentation states applicable licences.
Project documentation defines contributor license agreements, either by requesting a CLA or by stating the rights transferred to the project by a contributor not directly affiliated with the project.
Identity
Project/product/software has a distinct name regardless of its application area. A search by Google on the name plus keywords from the application area throws up the project web site in the first page of matches.
Project/software name or acronym is distinct within the application area.
Project/software name or acronym is easily discoverable, eg. the acronym is not a very common word or, if it is, the phrase provides additional information for discovery.
Project/product/software has a distinct name within its application area. A search by Google on the name plus keywords from the application area throws up the project web site in the first page of matches.
Project/product/software has a logo.
Project/product/software has its own domain name.
Project/product/software name does not throw up embarrassing “did you mean…” hits on Google.
Project/product/software name does not violate an existing trade-mark.
Project/product/software name is trade-marked.
form community
Web site has list of important partners and collaborators.
Documentation
[shifted from analysability]
Source code comments are written in an API document generation mark-up language (or language-specific style).
Documentation identifies the API generation took used.
Documentation describes how to build the documentation.
[ordered based on post-it activity]
Code is documented (comment blocks, etc.)
Pre/Post conditions are clearly defined
Comment blocks are well-structured and consistent, following language or project standards.
API documentation is autogenerated (e.g., by JavaDoc or Doxygen), and documents APIs completely (e.g., configuration files, property names, etc.).
Does not use terms like “intuitive”, “user friendly”, “easy to use”, “simple” or “obviously”, unless as part of quotes from satisfied users.
English language descriptions of commands and errors are provided but only to complement the above.
Plain-text files (e.g. READMEs) use indentation and underlining (e.g. === and ---) to structure the text, and avoid TAB character indentation.
Text files for higher-level documentation, such as README or CONTRIBUTING sections, use a common and identifiable markup language.
Further information is suitable for the level of the reader, for each class of user.
Explanations should be free of jargon.
Partitioned into sections for users, user-developers and developers (depending on the software).
States assumed background and expertise of the reader, for each class of user.
Documentation is on the project web site.
A link to the documentation is contained in the code. [PID should be used instead of URL]
Documentation is on the project website or the project website provides a clear link to an externally-hosted documentation web site. [PID over URL]
Documentation on the project web site makes it clear what version of the product the documentation applies to.
Documentation is held under version control alongside any code or other product components.
Where available, project takes advantage of public-facing tools' documentation support. For example, github supports a README file that uses Markdown. It's pretty quick to write and is displayed automatically when a person goes to get the source. Maybe this has more to do with 'buildability,' but in some cases it might be all the documentation an initial author writes.
The documentation for a specific software release is available.
Consists of clear, step-by-step instructions for use or adoption.
For common problems and error messages, the symptoms and step-by-step solutions are provided.
Documentation is updated to reflect new errors and resolution methods identified. [could be a FAQ - Y or N]
Gives examples of what the user can see at each step, e.g., screen shots or command-line excerpts; installation outcomes; working examples and error examples.
Lists resources for further information.
Provides a high-level overview of the product.
Constraints or restrictions are included in the documentation.
Quality control information is included in the documentation.
States command names and syntax, says what menus to use, lists parameters and error messages exactly as they appear or should be typed; or provides similarly explicit instructions on how to apply product.
Code or GUI examples provided in the documentation reflect the current state of the software, ie. parameter names in the code match those found in the documentation.
Uses teletype-style fonts for command line inputs and outputs, source code fragments, function names, class names etc.; uses appropriately styled fonts for key information and special terms or links.
Documentation follows commmon styles for displaying code blocks.
Documentation clearly distinguishes between input and output blocks.
For software, a best-practice guideline is selected and code is validated against it.
Is task-oriented or objective-oriented.
Testability
Suggested category grouping: project has unit and integration tests (1); tests are run by a continuous integration (CI) system once committed (2). Include coding standards and automated coverage analysis as (3) but less crucial than testing and CI. Reconsider criteria that are commonly provided by a CI system and whether those are suggested criteria on their own merit.
[add the linting concerns as sub group to testability. & rename testability.]
[guidelines for code style testing, replaces analysability]
Code is validated using a linter.
Code adheres to the style guide for the language or to the style guide provided for the project (or project's organization).
Code validation, related to style, is executed using automated tools (continuous integration tools or Coverity, for example).
A minimum test coverage level that must be met has been defined.
Continuous integration is supported – tests are automatically run whenever the source code changes.
For GUIs, project uses automated GUI test frameworks.
Project has automated tests to check conformance to coding standards.
Project has Unit Tests where apropriate and they can be run automatically, including by a CI system.
Project has automated tests to check test coverage.
Project has integration tests.
Project has integration tests and they can be run automatically, including by a CI system.
Project has scripts for testing scenarios that have not been automated (e.g. for testing GUIs).
Project has unit tests.
Project recommends tools to check conformance to coding standards.
Project recommends tools to check test coverage.
Project specifies how to set up external resources e.g. FTP servers, databases for tests.
Test results are e-mailed to a mailing list.
Test results are visible publicly.
Test results are visible to all developers/members.
Tests are automatically run nightly.
Tests create their configuration own files, database tables etc.
Tests create their own configuration files, database tables etc.
There is an automated test for this minimum test coverage level.
This e-mailing list can be subscribed to by anyone.
The test results e-mailing list can be subscribed to by anyone
Project measures both conditional and line test coverage levels
Expected test results are based on verifiable criteria.
Code is designed to support efficient testing.
Tests use mocking technology where appropriate.
Tests cover exceptional conditions as well as expected behavior.
Test suite includes tests to ensure correct permissions are maintained across the range of configurations.
For software implementing third party integrations, those integrations are tested in the test suite.
For software implementing third party integrations, the access needs of the integration for the software's stated purpose is clearly indicated to the user.
Open Source
Ed. note: suggest revisions for three situations - 1) project members are hosting their own repository management system (git, Trac, etc) which does affect public/private access and browsability; 2) project members are using a third party DVCS like GitHub or BitBucket which affects public/private but other aspects are not directly tied to the *project* assessment; c) project members post their software repositories to a domain/research code system which is more related to publication and preservation (but still about whether the code is accessible).
Merged accessibility & evolvability to capture activities for projects being managed as open source.
Ability to browse source code repository online.
Access to source code repository is available (whether for free, payment, registration).
Anonymous read-only access to source code repository.
Repository is hosted externally to a single organisation/institution in a sustainable third-party repository (e.g. SourceForge, GoogleCode, LaunchPad, GitHub) which will live beyond the lifetime of any current funding line.
All repositories developed by the project team are hosted externally in a sustainable thrid-party platform, either a code-hosting platform or a domain- or research-specific software repository.
Source distributions are available (whether for free, payment, registration).
Source distributions are available without the need for any registration or authorisation of access by the project.
Source distributions are freely available.
Binary distributions are available (whether for free, payment, registration).
Binary distributions are available without the need for any registration or authorisation of access by the project.
Binary distributions are freely available.
Downloads/versions page shows evidence of regular releases (e.g. six monthly, bi-weekly, etc.).
Web site describes end dates of current funding lines.
Web site describes how project is funded/sustained.
Web site describes project roadmap or plans or milestones (either on a web page or within a ticketing system).
Web site describes software requirements on hardware, operating system versions, tool versions.
Software includes no arbitrary shutdown dates.
[Ed. note: some criteria related to reuse of OS frameworks, ie. not reinventing every wheel. This is a placeholder and reminder.]
[Unmodified criteria]
Fitness
Accuracy
Tests exist to evaluate whether the results match the specification.
The precision presented in answers to the user is appropriate for the product's algorithm and implementation.
The product produces the same results in response to the same inputs (unless specification calls for randomness).
The product results are unaffected by increasing levels of use, analyzed inputs, or generated outputs; consistent with likely future demand.
The specific results provided by the product match what the specification calls for.
There are examples of outputs that follow the specification.
There is a specification of the algorithm against which results can be compared.
Credibility
All web interfaces use secure (https) protocol
Any group access privileges or functions are supported through group membership, not through group login via a single account.
Expertise of the originators of the project/product is represented throughout the content as appropriate (e.g., in author pages, references, citations, and about or background pages).
Opinions and perspectives are offered only as they relate to the mission of the project, and are clearly identified and put into context.
Presented information and data are consistent throughout the project.
Re-authentication (e.g., on lost password) requires appropriate verification (e.g., email to known account, or 2-factor authentication; not security questions).
The product user experience (e.g, response times) is materially unaffected by increasing levels of use, analyzed inputs, or generated outputs; consistent with user expectations.
User can access all appropriate capabilities and information, but can not access any content or feature that should be privileged.
Where registration is allowed, registration validation is consistent with required project security (e.g., simple captcha for insecure public-facing resources; email required for mildly resource-constrained software; manual identity verification for critical products)
Sustainability and Maintenance
Copyright
consider adding to section with licensing (not a high priority for EC)
Each source code file/web page/document has a copyright statement.
If supported by the language, each source code file has a copyright statement embedded within a constant.
If there are multiple web sites then these all state exactly the same copyright, licencing and authorship.
Web site states copyright.
Documentation states copyright.
Web site states key roles: who developed/develops the product, funds it, oversees the web site, etc.
Documentation states key roles: who developed/develops the product, funds it, oversees the web site, etc.
Analysability
[move doc statements to documentation (double-check), check learnability as well]
Source code is commented.
Source code is laid out and indented well.
Source code or content is structured into modules or packages.
Source code uses sensible class, package and variable names.
There are no TODOs in the code.
There is no commented out code.
There are no old source code files that should be handled by version control e.g. “SomeComponentOld.java”.
Coding standards are recommended by the project.
Coding standards, for each programming language used, are recommended by the project.
Coding standards are required to be observed.
Project-specific coding standards are consistent with community or generic coding standards (e.g. for C, Java, FORTRAN, Python, Ruby, etc.)
Coding standards are verified using a linter (should one be available for the given programming language).
Styleguides are provided for the project.
Source code comments are written in an API document generation mark-up language e.g. JavaDoc or Doxygen.
Project documentation identifies the API generation tool used.
Project documentation describes how to regenerate the documentation
Project uses automated documentation generation tool, e.g. Swagger or RAML.
Source code or content repository is a revision control system.
Source code or content repository is managed through a version control system.
Source or content releases are snapshots of the repository.
Releases follow protocols of the version control system in place.
Auto-generated source code is in separate directories from other source code.
How to regenerate the auto-generated source code is documented.
Project files for IDEs are provided.
Source code structure relates clearly to the architecture or design.
Structure of the source code or content repository and how this maps to the software’s components is documented.
Changeability
[not EC priority]
Changes in the source code repository are e-mailed to a mailing list.
Changes to the source code repository are made available through some notification system, whether mailing list of ticketing system notifications.
This e-mailing list can be subscribed to by anyone.
Anyone can subscribe to the notification system providing software release updates.
Contributions policy is publicly available.
Contributors retain copyright/IP of their contributions.
Project has defined a contributions policy.
Users, user-developers and developers who are not project members can contribute.
Contribution policy is not limited to code contributions and describes processes for anyone to contribute to the project even if nt project members.
Project has defined a stability/deprecation policy for components, APIs, pages, etc.
Releases document deprecated components/APIs in that release.
Releases document removed/changed components/APIs in that release.
A release is accompanied by a change log document (or update to a CHANGELOG document), including deprecated or modified components/APIs.
Stability/deprecation policy is publicly available.
Source code uses semantic versioning for major and minor releases.
Contribution policy indicates project's versioning method.
Community
update for new software publication practices (not paper necessarily) and in a better location (for ESIP)
Users are requested to cite the product if publishing papers based on results derived from the product.
Users are required to cite a boilerplate citation if publishing papers based on results derived from the product.
Web site has list of software that uses/bundles this product, or sites that reference this work.
Web site has list of the project’s publications.
Web site has list of third-party publications that cite the product.
Web site has quotes from satisfied users.
Web site has statement of number of users/developers/members.
Web site has success stories.
Developers exist who are not members of the project.
Users exist who are not members of the project.
Governance
[not EC priority]
Project has defined a governance policy.
Governance policy is publicly available.
Project website or code repository contains documentation describing the contribution policies and procedures.
Contribution policies and procedures included in a CONTRIBUTING file.
Contribution policies and procedures describes the code review process, including who performs the review, who accepts the reviews and how long reviews take.
Contribution policies and procedures clearly define decision making processes for all contributors.
Project website or code repository includes a code of conduct.
Portability
Ed. note: this entire section is flagged for review, to be resolved by either condensing the two main concepts (software can be built and run on multiple platforms and web application functions across multiple browsers) into more general criteria or by removing the section altogether and updating related criteria in other sections (such as buildability and documentation criteria related to that).
[not EC priority]
For code that compiles and does not use a VM: Uses a build system that interogates the build environment to determine features AND the software actually uses the result of that interogation
For code that is interpreted: Follows standard (likely ad hoc) practices for distribution and interpreter version/selection.
If the algorithm takes advantage of GPU architecture, is it hardware vendor agnostic?
Application can be built on and run under Debian.
Application can be built on and run under Fedora.
Application can be built on and run under MacOSX.
Application can be built on and run under RedHat.
Application can be built on and run under Solaris.
Application can be built on and run under Ubuntu.
Application can be built on and run under UNIX/Linux.
Application can be built on and run under Windows 7.
Application can be built on and run under Windows Vista.
Application can be built on and run under Windows XP.
Application can be built on and run under Windows.
Browser applications/sites run under Google Chrome.
Browser applications/sites run under Internet Explorer.
Browser applications/sites run under Mozilla Firefox.
Browser applications/sites run under Opera.
Browser applications/sites run under Safari.
Supportability
Ed. note: criteria marked as Grouping: Email need revision for the different modes of email support and regrouped accordingly (direct email, form-based contact, mailing list).
[not EC priority]
Above pages/windows/files describe, or link to, a description of “how to ask for help” e.g. cite version number, send transcript, error logs etc.
Project has an e-mail address.
Project has an e-mail address for support.
Project website includes a contact form for requesting support.
Project provides a mailing list for requesting support.
Project e-mail address has project domain name.
Project e-mail uses the project's domain name.
E-mail archives are publicly readable.
E-mail archives are searchable.
E-mail lists or forums, if present, have regular posts.
E-mails are archived.
E-mail archives or ticketing system shows that queries are responded to within a week (not necessarily fixed, but at least looked at and a decision taken as to their priority).
E-mails are read by more than one person.
If there is a blog, is it is regularly used.
Project resources are hosted externally to a single-organisation/institution in a sustainable third-party repository (e.g. SourceForge, GoogleCode, LaunchPad, GitHub) which will live beyond the lifetime of the current project.
Project resources are hosted externally to a single-organisation/institution in a sustainable third-party repository, such as GitHub, or a research software repository which will live beyond the lifetime of the current project.
Project has a ticketing system.
Ticketing system is publicly readable.
Ticketing system is searchable.
Software describes how to get support (in a README for command-line tools or a Help=>About window in a GUI).
Source code repository, in a README or similar document, describes how to get support.
Software, as built application, includes how to get support through a Help window or other common interface feature.
User doc has page describing how to get support.
Web site has page describing how to get support.
Web site has search facility.
Web site has site map or index.
Usability
Buildability
All mandatory third-party dependencies or external references are currently available.
All optional third-party dependencies or external references are currently available.
An automated build (e.g. Make, ANT, custom solution) is used to build any software.
An automated build (e.g. Make, ANT, Maven, Gradle, or other language-specific build tool) is used to build any software.
Dependency management is used to automatically download dependencies (e.g. ANT, Ivy, Maven or custom solution).
Source distributions have instructions for building corresponding software.
Code follows the GNU software guidlines and uses README, NEWS, INSTALL and COPYING files to describe the package (INSTALL includes the build information this case).
Source distributions list all third-party dependencies that are not bundled, along with web addresses, suitable versions, licences and whether these are mandatory or optional.
Tests are provided to verify the build or distribution has succeeded, or service/information is presented as intended.
Web site has instructions for building the product.
Web site lists all third-party dependencies that are not bundled, along with web addresses, suitable versions, licences and whether these are mandatory or optional.
Binary and source packages reference third-party code (including the acceptable versions) so that commonly-used tools (e.g., yum, apt-get) will automaticlly fetch those source/binary packages
Project uses build tools (e.g. automake, etc.) in a way that integrates with linux distro packager's expectations so they can/will build packages for public repos.
Project supports common/standard (ad hoc) binary packaing tools (e.g., RPM, python egg). This means that a 'buildable' package provides binaries (or the equivalent) so that it can simply be installed and does not have to be built!
Build instructions include platform-dependent details and answers to frequently asked questions.
Build instructions include clear version requirements for external dependencies (unless automated dependency management is provided).
Friendliness
All essential options and information for a task are visible, while excess information is avoided. Display elements are easily distinguished, and tool tips provided. Possible and impossible actions are indicated.
Display handles variety of scales, including mobile phone size if that is targeted, while allowing user ready access to needed controls.
The web application was developed using responsive design principles.
Generated data products are organized clearly, with embedded metadata wherever possible.
Generated data products are organized clearly, with associated metadata clearly identifiable.
Generated data products and their related metadata are provided in data packages and those data packages are organized clearly.
Lists of resources are neatly presented for easy access, comparison, access to details, and handling of scale. Both filter and scrolling are available; user can control number of items presented.
Simple and common tasks are easy to perform and communicated clearly and simply in the user's reference frame. The product minimizes the effort for users, by making appropriate assumptions, presenting appropriate defaults, and offering appropriate short-cuts.
System can be controlled or used by foreign speakers and color-vision-deficient individuals.
Project web site provides localization options, ie support for non-English speakers.
Project web site follows established guidelines for web accessibility.
Web application follows established guidelines for web accessibility whenever possible.
Terminology used throughout the system or product is internally consistent, appropriate to the context, and suitable for targeted users, avoiding codes unless universally known.
User commands are quickly and intuitively reflected in system behavior and subsequent system presentation (e.g., previous filters remain visible).
User intefaces maintain consistent patterns of behavior and style (e.g., buttons and cancel/save exits have consistent look and feel), creating an understandable presentation throughout the application.
User interfaces are organized in meaningful and useful ways that are recognizable to users.
User mistakes are tolerated gracefully; multiple levels of "command undo" are provided, paths to 'go back' are available, irreversible actions are rare but verified if needed, and suggestions are offered in response to faulty input.
User variation is handled automatically; varying sequences and input formats are handled, and reasonable actions are accepted.
Users are informed of relevant software actions, state changes, errors, and assumptions in clear and simple ways.
Installability
All GUIs contain a Help menu with commands to see the project name, web site, how/where to get help, version, date, licence and copyright (or where to find this information), location of entry point into user doc.
Web site lists all third-party dependencies and external references that are not bundled, along with (as appropriate) web addresses, suitable versions, licences and whether these are mandatory or optional. [this material must be contained with the software but not required in web site?]
Documentation lists all third-party dependencies and external references that are not bundled, along with (as appropriate) web addresses, suitable versions, licences and whether these are mandatory or optional.
All mandatory third-party dependencies or external references are currently available.
All optional third-party dependencies or external references are currently available.
Dependency management is used to automatically download dependencies (e.g. ANT, Ivy, Maven or custom solution).
Binary distributions list all third-party dependencies and external references that are not bundled, along with (as appropriate) web addresses, suitable versions, licences and whether these are mandatory or optional.
Binary distributions identify all third-party dependencies and external references that are not bundled, along with (as appropriate) web addresses, suitable versions, licences and whether these are mandatory or optional.
All other content distributed as an archive contains a README.TXT with project name, web site, nature, how /where to get help, version, and date.
All other content distributed as an archive contains a README, in a common plain text markup format, with project name, web site, nature, how /where to get help, version, and date.
All source and binary distributions contain a README.TXT with project name, web site, how/where to get help, version, date, licence and copyright (or where to find this information), location of entry point into user doc.
All source and binary distributions contain a README, in a common plain text markup format, with project name, web site, how/where to get help, version, date, licence and copyright (or where to find this information), location of entry point into user doc.
Binary distributions have instructions for installing the product.
Installers allow user to select where to install software.
Web site has instructions for installing or accessing the product.
When software is installed, or when a product or service has multiple facets, its contents are organised into sub-directories (e.g. docs for documentation, libs for dependent libraries) or IRL hierarchies (/about, /support, etc.) as appropriate.
When appropriate, installer integrates with external authentication sources (e.g., LDAP)
Installers are available for all relevant platforms (Windows, OSX, Linux, etc.)
Installer sets up necessary environment variables
Tests are provided (or silently performed) to verify the install has succeeded or service/product is fully available.
When an archive (e.g. TAR.GZ or ZIP) is unpacked, it creates a single directory with the files within. It does not spread its contents all over the current directory.
Uninstallers exist, and uninstall every file or warns user of any files that were not removed and where these are.
[Recommended installation is not through curl/wget to an uncontrolled resource.]
Learnability
A getting started guide is provided outlining a basic example of using the product.
API documentation is provided for userdevelopers and developers.
API documentation is provided for user-developers and developers.
Instructions are provided for many basic use cases.
Instructions are provided supporting all use cases.
Reference guides or contextual specifications (may be simple labels, if clear) are provided for all command line, GUI and configuration options.
Understandability
Architectural overview, with diagrams, is available.
Architectural overview, with diagrams, is included or referenced in the documentation.
Case studies of use are available.
Case studies of use are included or referenced in the documentation.
Descriptions of intended use cases are available.
Descriptions of intended use cases are included or referenced in the documentation.
Design rationale is available – why the system does things the way it does.
Design rationale is included or referenced in the documentation – why the system does things the way it does.
High-level description of how the product works is available.
High-level description of how the product works is included or referenced in the documentation.
High-level description of what the product does is available.
High-level description of what the product does is included or referenced in the documentation.
High-level description of what/who the product is for is available.
High-level description of what/who the product is for is included or referenced in the documentation.
Constraints or restrictions are archived with the source code.
Constraints or restrictions are included or referenced in the documentation and archived with the source code.
Other
Security
Suggested group ordering: Basic (1), Process (2), Tools (3).
Software is exposed to a static source code checker (e.g., Coverity)
Software authors use PKI (e.g., PGP) to sign releases (source and binary) and make the public key available on their website (and/or public key repos).
Authors are aware of and use public secure coding guidlines for their languages/technology. These include specific guidlines published by CERT and OWASP, cover both languages like C/C++, Java, Perl and technologies like Web services adn XSLT.
Projects have in place (and publicly visable) a security incident response plan.
Authors perform dynamic testing on their software
Authors work with an external organization that reviews/tests their software.
Sanitizes user input. (eg database apps sanitize for SQL injection, inputs that get displayed as HTML get filtered)
Web application defends against Cross-Site Request Forgery (CSRF)
Web application defends against Cross-Site Scripting (XSS)
File permissions of web-accessible files (eg User-uploaded files cannot overwrite or delete critical files)
User-uploaded files are not executable, cannot completely fill a disk,
Users cannot view arbitrary files through web app
Preservation/Archiving
Ed. note: review with documentation for duplication - make statement here for documentation archived with code and meets those doc standards.
Source code used to create data are archived with the data or separately with bidirectional references between the data and software.
Source code developed for accessing or using the data are archived with the data or separately with bidirectional references between the data and software.
Documentation are archived with the source code.
Licenses are archived with the source code.
Constraints or restrictions are archived with the source code.
Document describing known environment in which code was originally run
List known dependencies
consider container technologies to ensure reproducibility of software content or function.
documentation describing objectives of software, known limitations or bugs, and reason for and purpose of its existence
Ideally, preserved/archived code should include "test data", so future users can unpack and run code, and see if results appear as expected. (Unless data is already archived elsewhere)
Code should be documented in a natural language explaining each process/code section.
Archived code should not be an executable (.exe) but a group of files packaged together that can be built or reviewed by future users for understanding. Code should be as self-contained as possible.
Does the software use open source platforms and libraries, or does it require the use of licensed software?
Copyright/licensing must be noted and made clear to future users.
Archived code should include any documentation that applies to the particular release of the code--including both specific version documentation and overall software documentation.
Publication & Credit
Authors of software are identified in the suggested citation for the software.
Substantial contributions to software are considered authorship.
Other contributions to software are acknowledged in documentation.
Authors of documentation are identified in the suggested citation for the documentation.
Software used to create data are cited in the data documentation.
Data for which the software were created are cited in the software documentation.
Review committees consider and recognize software contributions and citations of software.
Publications about the data cite the software used to create the data.
Publications about the use of the data cite the software used to analyze the data.
Performance
Ed. note: Suggested new category to specifically address performance. Will need something regarding web app/service performance.
Rough order of time-complexity and memory-complexity are documented.
Is the algorithm parallelized? Or, does the algorithm scale across multiple CPU cores?
Does the algorithm use memory efficiently?
Performance requirements are documented. What is the requirement and why (for example, Page X should return in under 3 seconds b/c analytics show our major stakeholders navigate away after waiting for 3 seconds).
Performance limits are documented. (eg. software performance degrades when trying to analyze anything more than 1 million entities)
Notebooks
Ed. note: Suggested new category given the popularity of Jupyter/iPython Notebooks (or similar systems) in research settings.