Handling a working storage dictionary

What would you like to see in CSPro?
Forum rules
New release: CSPro 8.0
Post Reply
AriSilva
Posts: 591
Joined: July 22nd, 2016, 3:55 pm

Handling a working storage dictionary

Post by AriSilva »

Hi folks,
I´m using a working storage dictionary with a max = 9999, since I do not know in advance how many cases I should have.
It is a meny program used to control the data entry, calling the real data entry program after some specifications are defined.
The problem is that it takes too long to open the menu program (almost 2 minutes in a Samsung tablet that we are using), as I found out that the problem is related with this huge number (I changed it to 999 and the loading time dropped to close to 15 seconds).
This is a problem because this program has to be loaded for every new questionnaire (the data entry program calls back the menu program after finishing an interview).
What I think it is happening is that CsPro needs to initialize all the possible cases (9999) beforehand, even if it would not use all of them.
Would it be possible not to initialize all of them, and do it in a case by case situation, as needed? Or doing something like a "malloc" by chunks, let´s say, a 100 at a time, as needed?
Best
Ari
Best
Ari
aaronw
Posts: 561
Joined: June 9th, 2016, 9:38 am
Location: Washington, DC

Re: Handling a working storage dictionary

Post by aaronw »

I have run into this before where defining a large number of occurrences is slow regardless of whether they exist or not. I simply redesigned my application and I wonder if you can do the same. What are you loading into memory for each case? Also, why do you need an occurrence for each case in the working storage?
AriSilva
Posts: 591
Joined: July 22nd, 2016, 3:55 pm

Re: Handling a working storage dictionary

Post by AriSilva »

I need to have a list of all the interviews in the tablet, because the interviewer can reopen a saved partial (so I need a list of the partially saved ones), or s/he can open a previously closed interview because nobody was at home, etc, etc.
Of course I could read the database everytime I needed some info, but that would be much more time consuming.
Best
Ari
htuser
Posts: 631
Joined: December 19th, 2011, 6:26 pm
Location: Silver Spring Area, MD, USA

Re: Handling a working storage dictionary

Post by htuser »

@ Ari and Aaron,
What do you think if using sql query instead of the actual way (external/working storage dictionary)? You can store in sqlite (insert into, savepartial : Yes/No) and use sqlquery function to retrieve only where save partial=Yes. This will be more fast, simple and transparent.

Last year i was trying this way. And also, I think the sqlquery function will be more powerful soon since it will allow to query records, items in CSdb.
Best Regards,
G.VOLNY, a CSProuser from Haiti, since 2004
josh
Posts: 2399
Joined: May 5th, 2014, 12:49 pm
Location: Washington DC

Re: Handling a working storage dictionary

Post by josh »

In pretty much every CAPI application I've worked on we read the database every time to get the list of partial/completed/not started interviews. We do this using a forcase loop. The performance has been fine. Are you actually seeing a performance problem or are you assuming that there would be one? If it is actually slow we could probably help with some strategies to make it faster.
AriSilva
Posts: 591
Joined: July 22nd, 2016, 3:55 pm

Re: Handling a working storage dictionary

Post by AriSilva »

Answering both htuser and Josh´s posts:
1. My knowledge of sql is very limited (to say the least), maybe I should take a crash course (or read any SQL for Dummies book, do you recommend any?). So, I would rather prefer staying inside the CSpro ream
2. As for Josh´s questioning about the forcase loop reading the database every time I need a case, the first version of this program used that technique, and it was a little slow. I assumed (I did not do any benchmarking) it was because of the forcase loop. In any case, for this survey (we are doing a real operation in the field right now), I would not change the working storage strategy, but for the next ones I will consider Josh´s suggestions.

PS. I´m not joking about the sql issue, I really need to go into that, now that CsPro moved from .dat to .csdb, If you can send me any examples or anything related to that it would be much appreciated.
Best
Ari
Best
Ari
josh
Posts: 2399
Joined: May 5th, 2014, 12:49 pm
Location: Washington DC

Re: Handling a working storage dictionary

Post by josh »

I don't think SQL is the solution for this particular issue. I don't encourage people to SQL on the csdb file since we may change the structure of the database in future versions in which case your code would no longer work. It is better to stick to the CSPro functions when working with csdb files. The solution in this case is for us to make forcase on a big data file work faster and that is on our todo list.

There are however other uses for SQL in CSPro, particularly when it comes to working with paradata. I don't have any particular recommendations for SQL books/tutorials. I learned it from reading various sites online but it has been a while so I can't remember which ones.
Post Reply