When using mobile devices for a survey or census it is important to be able to transfer data collected in the field back to the head office. This allows for faster processing and analysis as well as better monitoring of the progress of the field operation.
In CSPro, transferring data between devices in the field and the head office is referred to as synchronization. CSPro supports data synchronization over the Internet between interviewers' devices in the field and a central server. Interviewers use CSEntry to collect data on tablets, phones, or laptops and then use the Internet to synchronize the data on their devices with a server at the head office or in the cloud. In CSPro, the data collection itself may be done entirely offline, with no Internet connection. When an interviewer is able to connect to the Internet they may synchronize with the server and transfer any data collected since the last synchronization. Synchronization may be done using Wi-Fi or a mobile data connection (2G/3G/4G).
Direct synchronization between interviewers and central server over the Internet
In situations where an interviewer rarely or never has access to the Internet, CSPro supports peer-to-peer synchronization between devices using Bluetooth. Bluetooth synchronization does not require an Internet connection. It is a direct connection between two devices that are in close proximity. Using Bluetooth, an interviewer may synchronize their device with a supervisor's device, transferring their data to the supervisor. Later, the supervisor travels to a location where they are able to connect their device to the Internet in order to synchronize with the central server. In this scenario, a supervisor might visit multiple interviewers in order to synchronize with them and later upload their data to the server at the head office.
Synchronization with supervisor over Bluetooth when interviewers do not have Internet access
CSPro supports both one-way and two-way data synchronization. This means that is possible to send data from the interviewer's device to the server as well as to download data from the server to the interviewer's device. You can configure your synchronization to only send data to the server, to only receive data from the server, or to both send and receive data.
It is possible to synchronize both CSPro data files and other non-data files such as application files, images, and text files. It is possible to update data entry applications in the field by downloading the latest
.pen and
.pff files from the server. This way, modifications to the application at the head office can easily be distributed to interviewers in the field. It is, however, simpler to deploy and update application files on mobile devices using the
Deploy Application tool. This tool packages application files for deployment, uploads them to a server and allows interviewers in the field to install and update them on a tablet by choosing
Add Application from the menu on their mobile device.
For synchronization over the Internet, a central server is required. CSPro supports three types of servers: CSWeb, Dropbox, and FTP.
- CSWeb: CSWeb is a web server running the CSPro synchronization server software. It is best for very large surveys and censuses. The server software is written in PHP and can be run on any web server that has PHP and the MySQL database software installed. The server can be set up on a computer at the head office that is connected to the Internet or it may be set up on a hosted website or virtual server in the cloud. Setting and maintaining a server for CSWeb requires knowledge and experience with web server maintainence and cyber security. Users that do not have these skills should consider using Dropbox or FTP instead. For more information about CSWeb servers, see the CSWeb help documentation.
- Dropbox: Dropbox is a free cloud based synchronization service. It is ideal for those doing small and medium survey operations. Dropbox requires no server setup or maintenance and avoids the cost and difficulty of setting up a CSWeb server.
- FTP: CSPro synchronization can be used with a FTP (file transfer protocol) server. For those doing small and medium survey operations who do not want to use Dropbox and are not able to configure a server to use CSWeb, FTP can be used for synchronization.
When synchronizing CSPro data files, synchronization is done at the case (questionnaire) level. CSPro keeps track of which cases have been added or updated and only sends cases that are new or have been modified since the last synchronization. This significantly reduces the amount of data transferred and therefore reduces bandwidth and the cost of air time. It also reduces the chance that two interviewers overwrite each other's work by both syncing to the same data file on the server. As long as two interviewers do not modify the same case at the same time, they may both synchronize to the same server without overwriting each other's data.
Once interviewers in the field have used data synchronization to upload data to the server, you can use the Data Viewer tool to download the data. When using CSWeb for data file synchronization, data is stored in a MySQL database on the server. When using either Dropbox or FTP, data will be stored on the server in a set of files in the directory
CSPro/DataSync. In both cases, the data on the server cannot be opened directly by CSPro. In order to retrieve the data that has been uploaded to the server, use the Data Viewer's
download function. This will download all the data on the server into a single CSPro data file that can be used by other CSPro applications and tools.
Basic synchronizations can be set up by specifying a few options
documented here.
For more advanced synchronization scenarios, CSPro has logic functions that can be used to implement data and file synchronizations. Using logic, it is possible to synchronize data files, including those associated with external dictionaries, and non-data files over either the Internet or Bluetooth.
There are several logic functions used for synchronization with a server:
- syncconnect is used to connect to the server and initiate the synchronization session.
- syncdisconnect is used after synchronizing to end the session.
- syncdata is used to synchronize cases in CSPro data files with the server.
- syncfile is used to upload/download non-data files to/from the server.
- syncparadata is used to synchronize paradata to/from the server.
- syncapp is used to update an application on the device from a new version on the server.
- syncmessage is used to send to and receive string messages from a Bluetooth server.
- synctime returns the time of the last synchronization of a data file or a specific case.
While it is possible to use
syncfile to synchronize CSPro data files, it is more efficient to use
syncdata since it will only send/receive cases that have been modified whereas
syncfile always sends the entire file.
You will generally call
syncconnect once to start the session followed by multiple calls to
syncdata,
syncapp, and
syncfile, followed by a call to
syncdisconnect. In the example below, we connect to the server, download the latest application files and upload cases from the data file associated with the data dictionary
SURVEY_DICT.
Bluetooth synchronization is done between two devices. One device, called the client, has logic similar to what is used for web synchronization. The second device plays the role of server. With Bluetooth synchronization, the server is passive. The device acting as the server runs only a single logic function,
syncserver, which waits and processes commands from the client device. Only the client device runs
syncconnect,
syncdisconnect,
syncdata,
syncfile,
syncparadata,
syncapp, and
syncmessage.
The client and server devices must be in close physical proximity in order to connect to each other. The server device must first execute the
syncserver function which causes it to wait for connections from nearby devices. Then the client device executes
syncconnect which initiates the connection between the two devices. The client then executes one or more calls to
syncdata and/or other sync functions, followed by a call to
syncdisconnect, which ends the session. This requires that the two operators using the devices coordinate to start the synchronization routines on the two devices at roughly the same time.
The following example shows the logic that would be written for the client and for the server:
Client Logic:
Server Logic:
// start peer-to-peer Bluetooth server
syncserver(Bluetooth);
The synchronization functions all assume that your CSPro application is already installed on the interviewers device. This means that it is not possible to use the above methods to download a new application for initial installation. While it is possible to simply copy the application to the interviewers device by connecting it to a PC via USB, in some cases it is preferable for the interviewer to download applications over the Internet. To download an application onto a device, use
Add Application from the menu in the
Entry Applications screen in CSEntry on an interviewer's mobile device. To package and upload an application to a server from where it can be downloaded, use the
Deploy Application tool. In addition, once an application has been installed on a mobile device using the
Deploy Application tool, it can easily be updated by re-deploying the application to the server and then using
Update Installed Applications from the menu on the mobile device or by using
syncapp in application logic.
Many errors that you may encounter during synchronization are caused by a lack of network connectivity rather than a problem within your CSPro program. An easy way to identify the source of the problem is to attempt to connect to the sync server within your web browser. For example, if you are trying to synchronize data on a tablet using Dropbox, go to https://dropbox.com within its web browser. If you are unable to connect then there is a problem with your connection to the Internet which needs to be resolved.
To get additional details on synchronization errors you can look at the sync.log file. This files lists all synchronization operations along with any errors encountered. This includes errors in uploading/downloading applications in Application Deployment and data download in Data Viewer.
On Android devices you can find the sync.log file in the CSEntry directory. Starting with CSPro version 7.5, the csentry directory is located at <external storage>/Android/data/gov.census.cspro.csentry/files/csentry. In earlier versions it was located at <external storage>/csentry. If you previously had an older version of CSPro on your Android device, the csentry will not be moved when you upgrade.
On Windows you can find the file in the directory %AppData%\CSPro. On Windows you can also locate the sync.log file from the Help menu in CSPro Designer under Troubleshooting. When asking for technical support with synchronization issues, please attach the sync.log file from your device to your email or forum post.