Problem concatenate several files

Discussions about tools to complement CSPro data processing
Forum rules
New release: CSPro 8.0
jberilo
Posts: 3
Joined: March 17th, 2012, 10:45 pm

Problem concatenate several files

Post by jberilo »

When I concatenate several files it repeats the header of the variables of any files in the file end using Cocatenate Tool. How can I solve this problem?
Gregory Martin
Posts: 1777
Joined: December 5th, 2011, 11:27 pm
Location: Washington, DC

Re: Problem concatenate several files

Post by Gregory Martin »

The concatenate tool was designed for combining CSPro data files, which don't contain headers, so the situation that you are faced with is slightly unusual. You could combine multiple files using a batch application, only writing out the header once, but this might be a lot of work for what you are trying to accomplish. See the attached application, and look at this logic:
PROC GLOBAL

numeric numFiles = 2;

array alpha (100) filenames(numFiles) = "file1.txt"
                                        
"file2.txt";

alpha (100) outputFilename = "output.txt";

alpha (300) str;

file inFile,outFile;


PROC NOREPEATEDHEADERS_FF

preproc

    
setfile(outFile,outputFilename,create);

    
numeric i;
    
    
do i = 1 while i <= numFiles
    
        
if setfile(inFile,filenames(i)) then
    
            
if fileread(inFile,str) and i = 1 then // write out the header for only the first file
                filewrite(outFile,strip(str));
            
endif;
            
            
while fileread(inFile,str) do
                
filewrite(outFile,strip(str));
            
enddo;
            
            
close(inFile);
            
        
endif;  
    
    
enddo;
    
    
close(outFile);
Attachments
noRepeatedHeaders.zip
(2.82 KiB) Downloaded 558 times
lls
Posts: 109
Joined: December 6th, 2011, 3:11 pm
Location: Geneva, Switzerland

Re: Problem concatenate several files

Post by lls »

Could you please share an example of a batch file to concatenate several data files if you have.
I would be very interested.
Thank you
Gregory Martin
Posts: 1777
Joined: December 5th, 2011, 11:27 pm
Location: Washington, DC

Re: Problem concatenate several files

Post by Gregory Martin »

The example above is a program that concatenates several files, only writing out the first line of a file once.

What kind of example are you looking for?
manishcspro
Posts: 67
Joined: June 26th, 2017, 2:15 pm

Re: Problem concatenate several files

Post by manishcspro »

I have two separate data files in .csdb format. when i concatenate these files, the cspro concatenate tool says :

Concatenation done with the following errors:

Skipping duplicate case 740203181922 in file C:\UPMPR17\inputdata\achdata\मासिक_प्रगति_मार्च_1819_मुरादाबाद मण्डल_09_08_2019_17_33_20.csdb. Previously found in file C:\UPMPR17\inputdata\achdata\मासिक_प्रगत

Skipping duplicate case 740203181923 in file C:\UPMPR17\inputdata\achdata\मासिक_प्रगति_मार्च_1819_मुरादाबाद मण्डल_09_08_2019_17_33_20.csdb. Previously found in file C:\UPMPR17\inputdata\achdata\मासिक_प्रगत

but both files have distinct id.

Please tell the reason and guide me how to resolve it.
Attachments
merge_achdata.zip
concatenate app
(2.47 KiB) Downloaded 227 times
josh
Posts: 2399
Joined: May 5th, 2014, 12:49 pm
Location: Washington DC

Re: Problem concatenate several files

Post by josh »

It is hard to say exactly what is going since your attachment does not contain the data files themselves. You will get the message about duplicate cases in two situations:

1) if between the two files you have two cases with the same case ids i.e. the same values for all of the id variables.
2) if between the two files you have two cases with the same internal id. This occurs if you created the case on one device and sent it another device either by sync or copying the file via USB and then modified the id variables on the second device. When you modify the id variables the internal id still the same so the concat tool treats it as a duplicate case.
manishcspro
Posts: 67
Joined: June 26th, 2017, 2:15 pm

Re: Problem concatenate several files

Post by manishcspro »

Thanks for response.Here are the two data files one of April (23 cases) and another of May(23 cases) but when concatenated is shows only 23 cases saying skipping duplicate cases id.But both files have distinct ids.Please guide me how to concatenate cases in this situation..Can i change myself internal case ids.
Attachments
datafiles.zip
(27.66 KiB) Downloaded 205 times
josh
Posts: 2399
Joined: May 5th, 2014, 12:49 pm
Location: Washington DC

Re: Problem concatenate several files

Post by josh »

The internal ids of the cases in the two files are all the same. I'm guessing you copied the data file for April and then modified the cases to create the file for May rather than creating a new file. To remove the internal ids you can convert the files to text format, which has no internal ids, and then concatenate the text files. If you want a csdb file in the end you can convert the concatenated file back to csdb although depending on what you want to do with the concatenated file that may not be needed.
manishcspro
Posts: 67
Joined: June 26th, 2017, 2:15 pm

Re: Problem concatenate several files

Post by manishcspro »

Thanks a lot..
manishcspro
Posts: 67
Joined: June 26th, 2017, 2:15 pm

Re: Problem concatenate several files

Post by manishcspro »

One more thing i want to do is to compare the data of a month from the previous month.I have created the pff application to generate report in html.For Example i want to display a column value of both the current month and the previous month at a time.For example based on the earlier attached data file i want to display a column value for May and April month side by side. Image of report is uploaded. Please help me to do the required.
Attachments
ach_image.jpg
ach_image.jpg (81.4 KiB) Viewed 5301 times
Post Reply