The first tool, Listing File Comparer, provides a way of quickly looking at the error percentages across a group of listing files. This is useful for people who process data, typically census data, on files split by geography.

The second tool, Save Array Viewer, is a program that visually displays the contents of save array files. This program is especially useful if you use DeckArrays for hotdeck imputation.

Also newly posted on the site is a tutorial about creating CAPI applications written by Anne Abelsæth of Statistics Norway: “Development of Data Entry and CAPI Applications in CSPro“

In this example, the main dictionary of the application is a junk dictionary. Any data entered to this dictionary will be ignored. We only use this dictionary to provide the framework for the main data entry application and as a way to enter a menu selection.

The external dictionary and form is actually where data is entered for this application. By using loadcase and writecase statements, it is possible to add and modify cases.

Whenever the data entry application is started, an array is populated with information about all of the cases in the data file. In this example, the program is hardcoded to expect that information about households 1-8 will eventually be added to the file. The program reports on what has been entered and what cases are remaining.

This list is created by using the find statement to check on all of the expected IDs in the external file (which really is the main data file for this application). Then the setvalueset and setcapturetype functions display the results on the screen.

Download the example here, or view the code below.

When converting checkbox variables into a single variable, there are two basic methods for correcting multiple checks: hotdeck between two marked boxes or determine an order of likelihood. If three or more boxes are marked, the edits become more complicated and, due to the small probability of such occurrences, one can either write complicated edits or the converted variable can be set to blank and edited later using more detailed content edits.

Before converting multiple checkbox variables into a single variable, one must first check the number of boxes marked. This can be done by counting the number of individual checkbox variables. Then the conversion coding will be done based on the number of checked boxes. If only one box is checked, the new variable can easily be created without any special methods to determine the value.

Assuming that the checkbox variables have a value of 1 if checked and 0 if not checked, one can count the number of marked boxes like so:

Using a Hotdeck

Hotdecking is a simple way to estimate the value of a variable based on the checked boxes and on related variables. In the example below, in cases when two boxes are checked, the selection of which of the two values will be used for the new variable is determined from the roofing material. The hotdeck array has two dimensions: the first for each combination of the two checked boxes, and the other for the roofing material. When there are no checked boxes, then the new variable is left blank, to be handled by later edits. When one box is checked, the new variable is coded according to the checked box and the hotdeck is updated based on each combination of two checked boxes and on the roofing material. If two boxes are checked, then the new variable is coded based on the hotdeck using the given combination of checked boxes and the roof type.

Question: Type of Dwelling?
Answers: Separate, Apartment, Joint/Barrack House
Conversion Method: Hotdeck based on roofing material

Order of Likelihood

In a series of responses, it may be decided that certain responses are more likely than others. When multiple boxes are checked, the predetermined more likely response will be recoded as the actual response. In the example below, enumerators may mistakenly select more than one source of drinking water if the household has more than one source. However, the enumerators were instructed to select only the most “modern” water source, so the responses were prioritized in the order that they appear, with the exception of tubewell. Thus, if tubewell and any other response were selected, the new variable is set to tubewell, and this recoding priority continues with tap, then well, and so on.

Question: Main Source of Drinking Water?
Answers: Tap, Tubewell/Deep Tubewell, Well, Pond, River/Ditch/Canal, Others
Conversion Method: Recode with priority (tubewell, tap, well, etc.)

If conversions like this are often being made, the code in the numChecks = 1 and numChecks = 2 sections can be combined in a reusable function.

The world of small-scale surveys may not change dramatically, but the impact of technology on censuses is huge. This is a photo from an East African country that recently conducted a census. The warehouse in this photo stores all of the census forms and requires many workers to operate:

What if an EA were not in the final set … how easy would it be to find in a mountain of forms? Imagine a world in which all data collection is conducted on a phone or a tablet and immediately sent to the operation headquarters. Data editing would be minimized and the time from collection to publication could effectively be cut to almost zero. Such a world will be an exciting one.

2012 should be a good year for CSPro, first with the addition of Unicode support, and then with steps towards the future of data entry, with focus on Android and other handheld development.

The CSPro help documents include some information on the export statement, but I often forget the syntax or what exactly the parameters should be. Fortunately, now with CSPro 4.1.002, a feature of the Export Data tool allows you to view the code that powers the export. By selecting Options -> Copy Logic to Clipboard, you can then take the export logic and insert it in the batch application of your choosing. This is a nice way to quickly get the basic export code needed and then you can build off of that.

For an example of a batch export that uses a lookup file, imagine that I have a data file describing stores and their customers. One of the attributes of a customer record is a country code that describes where the customer lives. When exporting the names of the customers, I do not want the two-letter country code, but the full name of the country. Using this list of country codes as a lookup file, I would go into the export tool, select the fields that I want to export, and then after copying the logic to the clipboard I would see code like this:

I would insert this code into a new batch application, add my lookup file code, and I would end up with something like this:

This feature in the Export Data tool vastly simplified my task, allowing me to focus on the lookup file programming, rather than the syntax of the export statement. My exported file now contains data from two files:

500  Pastry Pantry     Barack Obama          US  United States
500  Pastry Pantry     Angela Merkel         DE  Germany
500  Pastry Pantry     Hu Jintao             CN  China
800  Chocolate Heaven  Jacob Zuma            ZA  South Africa
800  Chocolate Heaven  Alexander Lukashenko  BY  Belarus

This is the input file:

10500Pastry Pantry
20500Barack Obama                                      US
20500Angela Merkel                                     DE
20500Hu Jintao                                         CN
10800Chocolate Heaven
20800Jacob Zuma                                        ZA
20800Alexander Lukashenko                              BY

Download this example.

To override the default text options, you must create a file called csentry.menu and place it either in the Program Files\CSPro 4.1\ folder, or in the folder where the PFF file for your application is located. This file has a format that is easy to follow. For example, to override the File menu text, you would place this text in the file:

File=Fichier
File_Open=Ouvrir une application
File_OpenDat=Ouvrir un fichier de données
File_Save=Sauvegarde partielle du questionnaire
File_Exit=Fermer

To add shortcut keys, place an ampersand before the shortcut letter. For example:

File=&Fichier
File_Open=&Ouvrir une application
File_OpenDat=Ouvrir un fichier de &données
File_Save=&Sauvegarde partielle du questionnaire
File_Exit=&Fermer

Placing the file in the Program Files\CSPro 4.1\ folder means that it will affect every single data entry application run on that machine. Alternatively, placing it in a folder with the PFF allows you to have different menus for different users.

Soon on this site I will post French and Spanish language menus on the Tools page. For now, if you would like to create your own menus, use this template file.

The function randomin works by accepting as an argument either an in list or a value set. The function then returns a random value that falls somewhere in the listed values. It is now easy to get a random value within a non-continuous range:

You can weight certain values by repeating the occurrence of the value in the in list. For example, the following example will return three times as many 1 values as 2 values:

The function is probably most useful when used with a value set. For example, if you have a data file for a survey and you are planning on adding a question to the survey about religion, you may want to test out your edits for this variable. Your old data does not have this variable, so you would write a batch edit program and then add random religion data to a test data file, which you could pass through your edit programs:

The randomizevs function is useful for data entry applications that use the extended controls introduced in CSPro 4.1. Sometimes you may want to ask questions and randomize the order of the possible responses to avoid bias due to the item positioning. In the preproc of the relevant question, you could execute the randomizevs function to present a different order each time:

In the above example, 98 might be a “None of the above” code and 99 might be a “Do not know” code. You do not want these codes randomized with the rest, you always want them at the bottom of the value set list, so you exclude them from the randomization.

The compiler may complain that versionNumber is not a function that exists. However, starting in CSPro 4.1.002, you can attach multiple logic files to an application. In this case, if versionNumber is defined in a different file, the code will compile successfully.

When compiling your code, CSPro will load any additional logic files as if the text of these files were inserted in PROC GLOBAL. This means that only things that can go in PROC GLOBAL can be declared in an external logic file, but that includes variables, arrays, and user-defined functions that you use across many applications. In the above example, I might create a file, myfunctions.app, that contains code that I use frequently. It would look like this:

There is no need to write PROC GLOBAL in this external logic file. To add the file to the application, I select File -> Add Files, and then select my External Logic File:

These files are listed in the tree in the Files window under the main logic file for your application. External logic files can be removed by selecting File -> Drop Files.

Download the above example as a batch application.

In the past, when using the count function it was necessary to specify what record or group you wanted to search through (in the above case, POP_REC). Now you can write the code without specifying this clause.

The seek function, new to CSPro 4.1, searches a record for the first instance of something being true. Now, in CSPro 4.1.002, you can search for the nth occurrence of the conditional statement. For example, this loop will continue until there are no longer two spouses in a household:

Finally, the sort function has long been a useful feature of the CSPro language, but now you can use it with a where statement. For example, a common request is to have a roster sorted based on relationships. This example sorts a roster by relationship, and then sorts the children in order by descending age: