Rote Loading

With this specific application, we can import data, but there are limitations due to its 2000-era handling of XML files.

  1. HTML forms uploading files have to…
    1. Have all the packets be received by the server.
    2. Process the file with the browser connection still open.
    3. The server has to tell the browser everything was received and is done.
  2. All this has to happen as one action within a 5 minute window.

A better method would allow just uploading the files to a page. Background processes would monitor that location and process the files independent of the browser. Notifications can be sent to alert the user the processing is done.

Or… recognize the echo XML file because you took too long and prevent it from being loaded or remove the data.

I ended up figuring out that if we split the files at about 5,000 records, then it should take about half the 5 minute window. I am pleased that for most I have seen that is true and about one in ten take so much longer that if I had cheated and gone with closer to the 5 minutes, I would be deleting duplicates. (This last is because some files are 50MB and others 100MB.)

The grumbling of this post is that I am on the 25th of 58 files. This is tedious. I am lamenting not creating a curl script to do this part for me. Automation is perfect for things like this.

 

The post Rote Loading appeared first on Rants, Raves, and Rhetoric v4.

From Rote Loading published April 14, 2017 at 01:04PM.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s