excel to cspro out of order file

[AriSilva]

Hi folks,
I´m having a problem converting an xls file to csdb. I´ve done that a zillion time, but when executing this this utility with an specific file it gives me an error message saying that the file is out of order, see below:
30598 cases were converted but some portions of cases were not output because the Excel data was not in order. Sort your data and try the conversion again. The cases out of order include:
• 1120004206604

In order to be able to report the problem, I´ve copied the original file, keeping only a few variables (including the keys), keeping also just the initial cases that may show the problem.
Some details that strike me:
1. The file is sorted.
2. The record pointed out as being out of order does not seem to be out of order.
3. I´ve made a small program to read the csdb file and list the cases. Comparing the list with the original file, the list has one record less than the original one, but the missing record is not related nor close to the “bad” record, its key is “11200204602202”, which does not seem to be out of order either.
I´m attaching the following files:
1. Input_Few.xlsx – input file. Both records, the “seemed guilty” and the one which will be missing are marked in yellow.
2. Dictionary.dcf – dictionary file
3. Xl2CsPro_Few.xl2cs – the converter
4. Output1.csv – The list comparing the input and output files, both records are also marked in yellow, showing missing one
Could you investigate it and tell what is the problem with the excel file?

[AriSilva]

By the way, version 8.0 works fine, no records out of order or lost.
And it is much much much faster.
Best
Ari

[Gregory Martin]

Thank you for the detailed bug report. I fixed the issue and we'll release CSPro 8.1.5 this Thursday.

In case you're curious, the issue actually wasn't with Excel to CSPro but instead was with some refactoring that I am doing to base CSPro code. I am changing a lot of code to use UTF-8, rather than wide-character, strings. This is a multiyear process that I'm doing in parts. There is a temporary conversion function that converts wide to UTF-8 strings, and one of these functions caches these conversions using a hash code.

Amazingly, two of your file's IDs, 1120004206604 and 1202046022020, had the same hash code, so when Excel to CSPro checked to see if 1202046022020 was already written out, instead of checking 1202046022020 it was checking 1120004206604, which is why you got the detail about the file being out of order.

I removed this optimization and now everything runs correctly. I'm very happy that you reported this issue before it caused more problems.