Introduction

Before you put a questionnaire live, it’s necessary to test the questionnaire. As a good practise, testing is done in the following steps:

  • testing during development

  • testing by one or more third persons, making sure to trigger all functionality in the questionnaire, including but not limited to:

    • each branch of each routing point

    • each filter condition with varying elements (eg in the case of ‘age=1 or gender=1’ you’d need to test both the age and the gender variable and their combinations)

    • calculations and scripts, in the context of relevant border cases (such as empty or inconsistent respondent input)

    • etc.

  • testing by running the random data generator

  • verifying the resulting data set

Note

A questionnaire is considered tested when these steps are performed. Additionally it is a good idea to do a so-called ‘soft launch’. A Soft launch is a phase where you invite a small group of people to participate in your survey, collect their feedback and again check their data. This way, when there is something unclear in phrasing, or when stuff is not working as you anticipated, the maximum damage to your data is the 5% or 10% of people who were in the soft launch.

The test screen

Introduction to the test screen

In the test screen you can generate a link that specifies how you want to test your questionnaire. You can use the configured test options in either of three ways:

  • You can directly click on ‘Start testing’ to try the generated test link

  • You can copy and paste the generated URL and share it with another tester

  • You can create a QR code that typically can be read by mobile devices

Options available in the test screen

Language

Pick the language you want to test. Make sure all translations for that language are present before testing.

Start at question

You can test a questionnaire by directly starting at a certain question or item. You can enter the label or the ordinal number of the item you want to start testing. Alternatively, you can just single-click on the item in the left-hand item list to have the system enter the right value for you.

Country

The use of the country variable is obsolete

Version

In case you have more than one version of your questionnaire, you can select here which version you want to test.

ExtID

The ExtID parameter typically can drive questionnaire logic and redirects, so depending on your setup, it could be relevant to add the ExtID value here.

Extra par

In case your questionnaire has self-built logic to capture incoming URL parameters, you can test them by adding them here.

Labels

This checkbox drives whether the labels of each question will be shown in the test screen. This might make the screen look somewhat different than the real live screen, but on the other hand it’s easier to see which question you are looking at during testing.

Test session

It’s recommended to leave this checkbox checked unless there’s a specific requirement to test ‘real completes’. Testing without this flag might result in real completes that need to be cleaned out of your data set manually at a later point in time.

No full quota

When this checkbox is set, no quota will drive any overquota logic during the session, so no matter what scenario, you will continue in the questionnaire even where a real respondent would be considered over quota.

Don’t use filters

When this checkbox is set, no filter logic will apply in the entire questionnaire; all questions, subjects and answer categories will be visible to the test-respondent.

Always show back button

In real fieldwork it might be desired to disable the ‘back’ button for respondents. However, having this ‘back’ button enabled might facilitate testing.

Note that your template needs some logic to support this option.

Warn before reaching complete status

It could be handy to avoid complete status, because after you reach complete status, you get redirected and there’s no ‘back’ available to test other scenario’s in the same participation. Checking this option shows you a screen before reaching complete status, so you can still use the ‘back’ button and test other answer configurations.

Maximum of one repetition

It could be tedious to go through many similar repeatblocks, while you only need to test one. Set this option to maximize the number of visible repeatblocks to one during your test session.

Ignore mandatory options

This option makes it possible to see and click through questions without giving actual answers.

Options to add to URL

In case you want to add your own parameters to the test-URL, here’s the place to do so.

Random Data Generator (RDG)

Introduction to RDG

The system can generate an export file for you that is filled with random data. This file enables you to inspect if the structure of your data is as expected. You can verify things like:

  • is the filtering of questions working correctly, in other words: is question X skipped correctly based on condition You

  • are all items where you expect data actually filled with data. This enables you to verify that routing is set up correctly

Warning

RDG is not a replacement for regular testing. It is an additional tool. Noteably, the browser interaction is not tested. That means that, should a question fail to display correctly in a browser due to a javascript error, this will not be noted by the RDG engine. Your data will look fine in the RDG dataset, even though in real life people would not be able to get past a certain question.

How to set it up

You can create an export with RDG data by setting this up in the export definition. Just select ‘Random cases’ and enter how many cases you want to generate. Note that at this moment 20 is the max. If you want to create cases only for a certain language, you can select the language in the box. Note that the random texts inserted by the system will not be in that language, but should you have built your questionnaire to contain language-specifi information, this logic will be followed.

RDGBias directives

Introduction to RDGBias directives

More often than not, the system will have a hard time simulating a respondent and getting the ‘qualified complete’ status. For instance, sometimes you want to allow only people to continue who use a specific product out of a range of 100 products. Another example is when only people with a certain zip code are allowed to continue, or there is other logic based on that zip code. This is where the RDGBias.txt file comes in.

The directives are rules you set in the definition of your RDG scenario. The directives are hints about which answers the RDG engine will pick. An example of a Bias content setup is below. Each line constists of an item- or variable name, followed by a TAB character, followed by a directive:

** A comment line is started with a star

** the following line will tell the system to just always pick 40 for variable 'age':
Age     40

** the following line will tell the system to select 1 for gender in 90% of the cases. In 10% of the cases it will select 2:
Gender 1*90,2*10

* the following code will tell the system to pick any of the zipcodes listed:
Zipcode 10001,10002,10003,10004,10005

* you can set the chance that a multiple response checkbox is set to a percentage:
ProductsOwned   50%

* you can set the chance for a specific multiple response checkbox to be selected more often:
ProductsOwned_10        90%

* you can use the special named variables 'xDontKnow' and 'xText':
ProductsOwnedDontKnow   90%

* and in case people select 'Other, specify', you can set values for the input box:
ProductsOwnedText       red,green*10,blue

* in case you're checking for flatliners/speeders, you can backdate the start time of the participations
* so the system calculates a longer completion-time:
StartTimeBackdate  120*5,300*5,500*5

* Note that assigning values to the ExtID system variable is also supported:
ExtID   Provider1*10,Provider2*10

* Note that for question type 'importance/satisfaction' the directives look like
* below, given item label 'ImpSat':
* The line below will make sure Importance is 1 in most cases for all subjects:
ImpSat_Importance       1*10,2,3

* The line below will make sure Importance for subject number 3 is fixed to 3:
ImpSat_3_Importance     3

Quotas in RDGBias directives

Additionally you can set initial quota values. Like with the RDG answers, setting initial values does not affect real values on the live survey, the values only apply to the RDG process. The syntax is like this:

QUOTA   Quotaname       123

You can set multiple values at once:

QUOTA   Quotaname1,Quotaname2   123

Or you can use wildcards:

QUOTA   *US*    123

This last directive will start the simulation with all quotas that match the mask US to 123

You can have multiple QUOTA directives in one RDG definition.

Query parameter RDGBias directives

Often times you want respondents to enter your questionnaire with certain information in the URL. For instance you might want to send the gender along in the query parameters. Below is an example of how to set query parameters in RDG:

REQUEST provider 1*10,2*90

The directive above will simulate a 10% chance that a respondent will enter the questionnaire with &provider=1 and a 90% chance they will enter with &provider=2.

You can have multiple REQUEST directives in one RDG definition.