Splitting XML documents over 10MB causes performance issues

We are about to go live with a CreateForm Director project that is going to be splitting a very large XML file to process student statements.

Bottom Line technologies indicated that there are performance issues when trying to split XML documents over 10MB. They indicated that the speed at which an XML document is split and processed slows down considerably as more and more records are processed. This has to do with the fact that when splitting the program has to read through the entire document for each split. The further you go into the file, the longer it takes to go back to the beginning of the file.

How many of you out there are using XML for your input file documents in CreateForm Director 6.2?

If you are using XML files, have you encountered this performance issue?

If you encountered this performance issue how much did the large files impact performance?

What we ended up doing here is we split our file into 3000 records per file which gave us an XML file size of about 5.5MB per file. We found that anything 6MB or over really slows down the response time.

We sent the results to TeamCain who forwarded the problem to Bottomline in hopes that they can correct the XML performance problem.

For now place only 3000 records per file is a work around for us. We want to ensure we stay well under 6MB per file (want an average of 5.5MB per file).