A Better File Upload Progress Bar using Python, AJAX Prototype, & JSON

The Python File Input CGI
About four months ago, Christopher Bottaro, another developer here at FineTooth, worked on a Python CGI script to handle file uploads and it is mighty ingenious. The CGI uses standard input as a stream, which PHP cannot do. The CGI writes the stream data as it arrives, in blocks, to a file buffer. This enables us to use another PHP server-side script to to poll the size of the file buffer while it’s being written to. With this, we can calculate the parameters necessary for a progress bar user interface (time remaining, speed of upload). We integrated the whole kit and kaboodle into our PHP Form class so that if a form indeed had a file input, the progress bar was all set up to popup automagically, and the form’s action was set to point to the CGI. Upon file upload completion, the CGI would perform a redirect to the ‘original’ action URI for data processing.

The Problem Case - File Inputs & JavaScript
As we’ve moved more and more towards using JavaScript to control our application’s form processing (as opposed to using the standard form action attribute as an URL) one problem situation has been caused by the file input. Since the file input is essentially an OS level object, JavaScript can’t access the information (thank goodness!) in order to send it programmatically to a handler. So what we’ve done in the past has been to use iframes for forms that needed file inputs. I’ve read that the Dojo Toolkit basically does this technique - by detecting if the form has a file input on it, and if so, it uses an iframe mechanism to submit the form. For awhile this was a good workaround. However, for aesthetic and programmatic design reasons, I wanted to come up with another solution.

Why not just use Iframes for those forms?
For one thing, iframes are just kind of annoying. If you want to access your JavaScript framework, you have to program in the scope of window.parent. Hopefully, the browser is caching your stylesheets and javascripts, but I’ve sometimes seen otherwise watching apache logs, so I don’t believe the iframe solution is always efficient. More importantly, though, using the classic form post technique and then rewriting output from the server to the iframe’s window is just not how I want to process forms in our application. I prefer to use Prototype’s Form.serialize and then Ajax.Request to deal with form transactions. Our application doesn’t have page refreshes overall, so introducing them into little iframes if and only when a form needed a file input just feels so cowardly. And, all that *really* needs to post is the file input data. So suddenly, the brain dinger started going off.

Embedding N-Iframes
Instead of using an iframe for the entirety of the form, I wanted to try embedding iframes individually for each file input that needed to be on the form. I could have an iframe that loaded up its own form object into its window.body with only the file input and some session information. Then, via the controlling form, we can use javascript to submit each embedded iframe’s form programmatically. Once the poller running in the main form detects that all of the files are safely on the server, we can submit the rest of the form data using Ajax.Request along with some logic to capture the information about the freshly uploaded files. It sounded funky, but lo-and-behold, it works and it scales.

A diagram might help make more sense of this approach:

Form *text input *text input *Iframe *form *file input *text input etc...

What’s key here is that instead of having a single CGI try to deal with multiple file inputs and all the other inputs on the form, our CGI is doing only what we originally wanted it for in the first place - the file upload.

Here’s the form processing logic:

* User chooses file(s) to upload in the form. * User presses submit. * JavaScript validation runs against the form. * JavaScript looks inside the form for any embedded iframes. * If it finds them, it submits the embedded form to the CGI. * An interval runs again and again until all the files are reporting 100% upload * The rest of the form data is at last sent using Ajax.Request

A few other niceties
Since our CGI is writing data in chunks, and since the juciest information about the file is in the first little bit of data, the CGI can do a little regular expression matching early on in the process to capture the file’s mimetype and name. Our poller can then read this information and we can show a purty little icon for the filetype and also reference the actual file name in the poller window - making it look pretty slick!

The Poller
The return data from the poller comes back to the browser as JSON and looks like this:

({"success":true,
"percent_done":3,
"kps":3,
"timeleft":"21 mins 8 secs",
"filename":"01 Mistakes.mp3",
"mimetype":"audio\/mpeg",
"mimetype_iconsrc":"music.png",
"current_size":126676,
"total_size":4044948})


So Why can’t we do multiple files all at once?
Originally I wanted a display that resembled Firefox’s Dowload Manager. One problem I haven’t found an adequate answer to is, why does it seem that I can’t be sending up multiple CGI requests from the embedded iframes at once? I can see that apache seems to be blocking until all but the last CGI are left in terms of sending responses back to the browser. Maybe it is the browser doing that; I did find one mention of that being the case back in old versions of Netscape, but I can’t seem to find any other validation of this limitation anywhere. So until that issue is resolved, files will be sent and polled sequentially in this version.