Changes during data entry phase

Discussions about CSEntry
Forum rules
New release: CSPro 8.0
Post Reply
Floris

Changes during data entry phase

Post by Floris »

I have done a survey with 400 households, and I'm about to start the data entry after having designed my CsPro application. Now, I can imagine that somewhere along the way (e.g., after having entered 100 cases), I might still discover a mistake in the application. I would fix it and enter the remaining cases with the fixed version (in practice, a new .pen file).

However, I was wondering how this is reflected in the file in which the data are saved. It seems that you can continue using the same file, but how does that file "notice" and "adapts to" the changes? E.g. suppose I add a new variable, is this variable then included in this already existing data file, and what happens with the 100 records already existing?

In short: does the save file adapt to application changes throughout the data entry process, and if so, how?

Many thanks!
josh
Posts: 2399
Joined: May 5th, 2014, 12:49 pm
Location: Washington DC

Re: Changes during data entry phase

Post by josh »

You can update your application after having already entered data however, if you modify the data dictionary you have to be careful. The data dictionary specifies the start positions and lengths of all the variables in your data file. If you modify the start position or length of a variable in the dictionary then when you try to open data entered with the old version your dictionary, the data items will not be read properly since the new version of your application will be looking for items in the new start position and with the new lengths.

To avoid any problems always enter new variables at the end of a dictionary record instead of at the start or in the middle. This way they will not cause the start positions of any of the preceding items in the record to change. If you only add items to the end of the record you will be able to read old data files with the new dictionary. The new items will be blank in the old data files but the other items will be unchanged.

If for some reason you must insert items at the start or middle of a record then you can use the reformat data tool to fix the old data files to work with the modified dictionary but I find in practice it is much simpler to just add the items to the end of the record and avoid reformatting. Note that even if a variable is at the end of the record in the dictionary you can still have it appear in whatever order you want on the data entry form by simply moving the item in the form tree.
bmuganda
Posts: 37
Joined: January 18th, 2018, 11:57 am

Re: Changes during data entry phase

Post by bmuganda »

Hi, Josh,

I have experienced the same especially where data especially where data has been collected with a particular field size, then somewhere along the line the field length has to change to accommodate more characters. When you download the data some fields seem to have shifted. When you perform reformatting on the data, it now corrects the shifted fields and shifts the previously corrected well aligned fields.

Is there any particular way of realigning all this fields of different field length into one standard format?

Regards
Gregory Martin
Posts: 1777
Joined: December 5th, 2011, 11:27 pm
Location: Washington, DC

Re: Changes during data entry phase

Post by Gregory Martin »

This is one of the challenges of changing the field lengths during data collection, especially if you don't reformat the existing data at the time. Hopefully you have some way of knowing when the data was collected, like an INTERVIEW_DATE field in your application. If you have something like that, you could write a batch application that would separate the old data from the new data. For example, to write out the old date, I might write this:
if INTERVIEW_DATE >= 20190501 then
    skip case
;
endif;
And to get the new data:
if INTERVIEW_DATE < 20190501 then
    skip case
;
endif;
Then I could reformat the old data and concatenate it with the new data.
Post Reply