Beta testing

5.1 What is beta testing for?

Testing your project and making iterative changes based on feedback from others is vital to a successful crowdsourced transcription project. By soliciting and reviewing feedback on your project’s Workflow, interface, and Subject Sets, you will ensure that volunteers from diverse backgrounds and levels of proficiency are able to effectively contribute to your project.

Beta testing is a required phase of all publicly launched and listed Zooniverse projects, in which a set of volunteers are asked to test your project Workflow(s) and provide feedback on their experience. This is a formal process facilitated by the Zooniverse team.

Prior to formal beta testing, you should also test your project with friends or family who are trying it for the first time—and watch them as they do it! See where they struggle, ask them for feedback, and review the data they submit.

As you receive feedback and implement changes, you will iterate through different versions of your project. This will help you produce a Workflow that balances both an intuitive and enjoyable volunteer experience and the best quality data.

5.1.1 The project review process

In order to launch your project publicly and be listed on the Zooniverse project page, you will need to go through project review, which includes an internal review by the Zooniverse team as well as beta review by Zooniverse volunteers.

Before you apply for review, you should create a beta testing Subject Set and upload it specifically for this purpose. For Transcription Task projects, we recommend creating a beta testing set with around 25 Subjects, depending on how much text each image contains. The aim is to include enough data to last for five to seven days of beta testing, but not so much data that Subjects don’t get retired. That way, your beta test will produce enough completed classifications to be able to test your aggregation and data analysis methods. Your beta test set will need to be linked to an active Workflow.

In addition, you will need to ensure that your project page, including the About, Team and FAQ sections, contain enough information for your reviewers to gain a sense of your project. Additionally, your project must have a Tutorial, Field Guide and Help text, and your Talk boards must be set up. Apply for review through the ‘Visibility’ tab of the Project Builder. The most up-to-date requirements will be listed there.

Before we send your project to our volunteers for beta testing, your project will first go through the internal review process, in which a member of the Zooniverse team will review your project based on specific criteria. Your internal review feedback will be delivered via a private Talk board, which your internal reviewer will set up. Make sure you are subscribed to Talk email notifications, so you don’t miss this notification and delay your review process.

The project will then be sent to Zooniverse volunteers who have signed up to be beta testers. They will submit beta classifications to your project as well as provide feedback through a standard Google form which is set up for you by the Zooniverse team and linked on your project via the Announcement Banner.

Please note that this process may take some time to complete. Based on past experience, the entire process can vary from eight-to-sixteen weeks from applying to beta to full launch.

5.1.2 When to apply for review

It is important to go through the review process early enough to have enough time to implement any needed changes to both your Subject Sets and your Workflows. You should spend time reviewing and editing your project with contacts, colleagues, or friends so that you can go into the review process with a fairly robust project.

Depending on the feedback you receive from your internal review and your beta testing, you may elect to go through another round of testing with beta testers after implementing changes, especially if those changes result in a significantly different Workflow requiring different data analysis or quality control methods. You may also decide to test out new Workflows after your project has already launched, though additional beta tests are not required to add new Workflows to a project after it has launched.

5.1.3 What feedback is good feedback?

Unfortunately, you will never be able to please everyone, and beta testing is no exception. Much of the feedback you receive will be invaluable, but some of it will be about things you cannot change.

For example, if you have asked volunteers to transcribe an 18th-century text, helpful feedback may be to include examples of obsolete abbreviations in the Field Guide. Feedback that is less helpful may be that the handwriting is difficult to read. While this is good to know, in that it will help you better prepare volunteers for engaging with difficult scripts, it is also not something that can be resolved through editing your project Workflow.

Be careful not to overfit your post-beta edits to a small number of people; make sure that you're considering the full scope of the feedback you receive, including those that are in contrast with one another. If one person says they aren't interested in your project, but 10 people say they can't wait to participate, don't waste time trying to appeal to one person.

5.2 Beta: step by step

If you want your project to launch publicly, you must go through the review process. A public launch means that 1) we will send out a launch newsletter to our full volunteer community; and 2) your project will be marked as an ‘official’ Zooniverse project and listed at zooniverse.org/projects. If you do not want to launch publicly, you do not have to apply for beta review. Just build a project, set it to public, and send out your URL to your audiences. Please note that choosing to run a private project (and therefore skipping beta review) may limit our ability to assist you in later project stages – beta testing is a great way to catch issues with your project early on, while they are still easy to change.

Read our overview of the project review process for additional details.

5.2.1 Beta checklist

When you are ready for beta review, go through the checklist found in the Visibility tab of the Project Builder. At this stage, your project should be ready for volunteers to use – don’t submit something for beta testing that is unfinished, or that you wouldn’t send to the public.

5.2.2 Internal review

When you initially apply for review, your project will first be reviewed by a member of the Zooniverse team. The team member will deliver their feedback using our standard internal review form. Once they have finished their review, the team member will write to you via a Talk post on a private (Team + Zooniverse) board they set up as part of the internal review process. Make sure you are signed up to receive Talk emails for your project, or you may miss this step. Once you have read the feedback, you can respond to the reviewer via the review form. Note that there is also a section for you to fill out, too.

5.2.3 Beta review

After you have passed the internal review, it is time to schedule the beta review. Every Tuesday we send projects to the Zooniverse beta participant community for review. The max we send out on any given Tuesday is 3 projects. Your internal reviewer will tell you the next available slot and give you the opportunity to delay if needed. Beta lasts for about 1 week, and you will receive a link to your feedback via your private Talk board. Once the beta test is over, it is your responsibility to review your feedback and make any necessary changes.

As you prepare for beta, be sure to look at an example of the Feedback Form we will send to our beta testers.