error 500 when synchronizing

Discussions about syncing data via Bluetooth, Dropbox, FTP, and using CSWeb
Forum rules
New release: CSPro 8.0
AriSilva
Posts: 591
Joined: July 22nd, 2016, 3:55 pm

error 500 when synchronizing

Post by AriSilva »

Hi folks,
I´m getting this 500 error when using the syncconect and syncdata(get). The database has 8354 cases and the download always stops after downloading 3100 cases.
At the beginning of the survey we were able to download much more than 3100 cases, 5000, 6000 etc, but now it stops at the 3100 limit.
When using the pff directly reading the csweb74 server with apptype=sync and syncdirection=get it works MOST of the times, but not always, I´ve already got this 500 error using this pff, but it is much better than the syncconnect function.
Best
Ari
josh
Posts: 2399
Joined: May 5th, 2014, 12:49 pm
Location: Washington DC

Re: error 500 when synchronizing

Post by josh »

500 is a general http code for an internal server error. It could be a lot of different things. We would need to see the sync.log from the client and the api.log from the server to know what is going on.
AriSilva
Posts: 591
Joined: July 22nd, 2016, 3:55 pm

Re: error 500 when synchronizing

Post by AriSilva »

Thanks, Josh,
I´ll try to force the error again and look at the logs.
Best
Ari
AriSilva
Posts: 591
Joined: July 22nd, 2016, 3:55 pm

Re: error 500 when synchronizing

Post by AriSilva »

Hi Josh.
Attached the 2 log files with the 500 error.
Attachments
api.zip
(529.62 KiB) Downloaded 191 times
Best
Ari
josh
Posts: 2399
Joined: May 5th, 2014, 12:49 pm
Location: Washington DC

Re: error 500 when synchronizing

Post by josh »

I see this error in the api.log a few times:

[2020-07-29 12:56:55] php.CRITICAL: Fatal Error: Allowed memory size of 67108864 bytes exhausted (tried to allocate 4593380 bytes) {"exception":"[object] (Symfony\\Component\\Debug\\Exception\\OutOfMemoryException(code: 0): Error: Allowed memory size of 67108864 bytes exhausted (tried to allocate 4593380 bytes) at /home/clvcombr/public_html/csweb74/src/AppBundle/CSPro/UploadCasesJsonListener.php:255)"} []

I'm not sure why CSWeb wants to allocate that much memory. Is your questionnaire long?

The easy fix is to have your system admin increase the php memory limit to be more than 67mb. Maybe 128mb.
AriSilva
Posts: 591
Joined: July 22nd, 2016, 3:55 pm

Re: error 500 when synchronizing

Post by AriSilva »

Thanks, Josh,
We´ll try that fix increasing the memory limit.
The questionnaire is not that long, almost 2k bytes, and so far we have 9k cases.
The interesting thing it is that, when it crashes, it is always at the 3100 case.
Best
Ari
josh
Posts: 2399
Joined: May 5th, 2014, 12:49 pm
Location: Washington DC

Re: error 500 when synchronizing

Post by josh »

The algorithm starts by sending the first 100 cases. If the server handles that chunk of 100 cases fast and without error then it tries 200, then 400... There is an upper limit on the size of data we send but it may be set too big. We stop doubling if it takes more than 30 seconds to upload the chunk of cases or if the size of the data is greater than 10mb or it contains more than 100,000 cases. The idea is to send large numbers of cases when you have a fast network connection (like when you are at the office) and smaller chunks when you have a slow connection (like a tablet in the field). That works well on my server even with files that contain a million plus cases but I'm guessing the memory limit on my server is set higher than it is on yours. We will revisit the upper limit but it will be quicker for you to increase the memory limit on your server.
AriSilva
Posts: 591
Joined: July 22nd, 2016, 3:55 pm

Re: error 500 when synchronizing

Post by AriSilva »

Thanks Josh,
Increasing the memory limit solved the problem.
But, I´m not sure that this is a coincidence, after this change some uploads from the field are not being executed, hat is, the server database is not updated accordingly.
Some interviews were supposed to be finished (and we see them as finished by looking in the interviewer´s local database), but this is not reflected in the server db.
How is that possible?
I´ve made some tests on my machine, and in my case the interviews are updated.
I really cannot understand what is happening.
Best
Ari
aaronw
Posts: 561
Joined: June 9th, 2016, 9:38 am
Location: Washington, DC

Re: error 500 when synchronizing

Post by aaronw »

CSPro will only synchronize new data. For some reason the server's history has a new revision than the client. You can force the interviewer's data to upload by deleting their sync history. It's always a good idea to back up your data before doing something like this.

You'll need to copy the interviewer's CSDB file to your computer. Open it in DB Browser for SQLite, select "Browse Data", sync_history. Now copy the CSDB file back to the interviewer's tablet and have them resync. All data will be sent up and they should be in a good state now.
AriSilva
Posts: 591
Joined: July 22nd, 2016, 3:55 pm

Re: error 500 when synchronizing

Post by AriSilva »

Thanks, Aaron, got it.
What I did not understand in the step by step instruction you´ve written is that after selecting browse data, sync history, shouldn´t I do something with the history (maybe delete it), or just browse through it should be enough?
In order to do that procedure in every interviewer´s database, wouldn´t it be better to change the revision date in the server, if this is at all possible?
I do not trust the interviewers to do that themselves, so, maybe the best approach, as we are executing as CATI survey using desktops and notebooks (csentry in windows10), to do something like a Teamviewer with them and somebody from my central staff execute that procedure?
Best
Ari
Post Reply