jumbled .not data

Discussions about tools to complement CSPro data processing
Forum rules
New release: CSPro 8.0
Post Reply
mmhill
Posts: 18
Joined: August 5th, 2014, 3:26 pm

jumbled .not data

Post by mmhill »

Though we now use v7 the following problem is about the results of v6 data entry. We are going back to work on Notes fields entries with the eventual goal of combining with v7 notes. The following bullets summarize our understanding of the situation:

1. CSPro allows attaching long text fields (NOTES) to variables requiring additional information.
2. Older versions of CSPro produced NOTES data in text files with extension .NOT. Newer versions produce .csnot text files. The latter do not seem to jumble lines of text like the former.
3. CSPro does NOT permit importing data collected in older, .not producing, versions of CSPro into newer, .csnot producing, versions.
4. The .not files present data in a manner that challenges programmatic efforts to import them into SAS.
a. The major difficulty is that the lines appear jumbled.
b. They are inconsistently formatted; all start with the ID, but some follow that ID with the variable name, while others follow the ID with a fragment of the notes.
c. Files often contain repeated lines, which are not difficult to remove.
d. It has been our experience that the pattern that exists for one NOTES file or collection of notes does NOT remain consistent across time, files, or entries. In other words, a pattern for one entry does not necessarily apply to other entries.
e. Typically the .not fields contain continuations of text appearing in variable fields present in the data entry form; consequently, they often start abruptly. So, the import program needs to know at which line to start.

A more frequent issue is the lack of import re: the supplemental notes fields (data entered using the Ctrl+N function in CSPro). When this occurs, the “main” Notes content is typically imported w/o issue, but the supplemental content does not. This sounds like a different problem but want to mention in case related.

Thank you.
-MJ Hill
Gregory Martin
Posts: 1777
Joined: December 5th, 2011, 11:27 pm
Location: Washington, DC

Re: jumbled .not data

Post by Gregory Martin »

Hey Mary Jane,

Addressing some of your points:

2) The original .not file did wrap the notes, which drove me crazy because as you mention, it made it quite difficult to parse notes. We changed things with the .csnot file to make it easier to work with notes.

3) If you open a text data file in CSPro 7.x and no .csnot file exists but a .not file does exist, CSPro will convert the notes and you should end up with two files: .not (the old notes), .csnot (the new notes). Once they both exist, the old notes will be ignored. In CSPro 7.0-7.3, the text notes were only read in data entry applications, so the notes wouldn't get converted if you opened the file in batch, for instance. In CSPro 7.4, text notes can be accessed in batch applications as well.

The old notes format was fixed width:

- key (case IDs)
- 32 characters for the field name
- 17 characters for the occurrences (5 digit occurrences with periods separating them)
- the note content, which could be wrapped; when wrapping notes, the first part of the line (key+name+occurrences) would be repeated

The new format is also fixed width but isn't wrapped. It's described here:

https://www.csprousers.org/help/CSPro/n ... csnot.html

Working with notes is more convenient than ever using CSPro 7.4. If you open a data file in Data Viewer, you can select File -> Save As -> Notes. That will create a data file containing just the notes along with a dictionary that you can use to process/export the notes.
mmhill
Posts: 18
Joined: August 5th, 2014, 3:26 pm

Re: jumbled .not data

Post by mmhill »

Hi Greg-
Thanks for your usual great help!!!
-Mary Jane
Post Reply