Emscripten supports lazy loading of binary data from HTTP servers using XHR. This functionality can be used to create a backend for synchronous file access from compiled code.
The backend can improve start up time as the whole file system does not need to be preloaded before compiled code is run. It can also be very efficient if the web server supports byte serving — in this case Emscripten can just read the parts of files that are actually needed.
Warning
This mechanism is only possible in Web Workers (due to browser limitations).
Note
If byte serving is not supported then Emscripten will have to load the whole file (however big) even if a single byte is read.
An example of how to implement a synchronous virtual XHR backed file system is provided in the test code at tests/test_browser.py (see test_chunked_synchronous_xhr). The test case also contains an HTTP server (see test_chunked_synchronous_xhr_server) showing CORS headers that might need to be set (if the resources are hosted from the same domain Emscripten runs from, there is no issue).
The tests use checksummer.c as the Emscripten-compiled program. This is simply a vanilla C program using synchronous libc file system calls like fopen(), fread(), fclose() etc.
JavaScript code is added (using emcc‘s pre-js option) to map the file system calls in checksummer.c to a file in the virtual file system. This file is created early in Emscripten initialisation using FS.createLazyFile(), but only loaded with content from the server when the file is first accessed by compiled code. The added JavaScript code also sets up communication between the web worker and the main thread.
You will need to add JavaScript to the generated code to map the file accessed by your compiled native code and the server.
The test code simply creates a file in the virtual file system using FS.createLazyFile() and sets the compiled code to use the same file (/bigfile):
if (typeof(Module) === "undefined") Module = {};
Module["arguments"] = ["/bigfile"];
Module["preInit"] = function() {
FS.createLazyFile('/', "bigfile", "http://localhost:11111/bogus_file_path", true, false);
};
Note
The added JavaScript should also include code to allow the web worker to communicate with the original thread.
The test code adds the following JavaScript to the web worker for this purpose. It uses postMessage() to send it’s stdout back to the main thread.
Module["print"] = function(s) { self.postMessage({channel: "stdout", line: s}); };
Module["stderr"] = function(s) { self.postMessage({channel: "stderr", char: s, trace: ((doTrace && s === 10) ? new Error().stack : null)}); doTrace = false; };
Note
If you use the above solution, the parent page should probably contain handwritten glue code to handle the stdout data.
You will need a page that spawns the web worker.
The test code that does this is shown below:
<!doctype html>
<html>
<head><meta charset="utf-8"><title>Chunked XHR</title></head>
<html>
<body>
Chunked XHR Web Worker Test
<script>
var worker = new Worker(""" + json.dumps(worker_filename) + r""");
var buffer = [];
worker.onmessage = function(event) {
if (event.data.channel === "stdout") {
var xhr = new XMLHttpRequest();
xhr.open('GET', 'http://localhost:8888/report_result?' + event.data.line);
xhr.send();
setTimeout(function() { window.close() }, 1000);
} else {
if (event.data.trace) event.data.trace.split("\n").map(function(v) { console.error(v); });
if (event.data.line) {
console.error(event.data.line);
} else {
var v = event.data.char;
if (v == 10) {
var line = buffer.splice(0);
console.error(line = line.map(function(charCode){return String.fromCharCode(charCode);}).join(''));
} else {
buffer.push(v);
}
}
}
};
</script>
</body>
</html>
""")