Interactive Dynamic Pipelines

In the previous section we showed how to sequence a pipeline using the DPML language. In this section we show how to create dynamic pipelines. We also show how NetKernel manages exception handling within a pipeline and how we can use the NetKernel debugger to set break points in a pipeline.

Interactive

In this pipeline we will parameterize the search term so that we can interactively find speeches by Gloucester containing any word we like. Before we investigate how this parameterized pipeline operates you might like to try it!

Below is the interactive DPML process which we just ran (note: you might prefer to inspect exactly the same process written in a more traditional 'procedural style' using Beanshell (scripted Java) - you'll find it here)

Any DPML process (or indeed, any service written in any language on NetKernel) may receive external parameters. In the example above an external parameter, called param and referenced with the URI this:param:param, is received by the process. In this demonstration the parameter is a name-value pair XML document containing the HTTP GET parameters, it was created by the HTTP bridge service and issued to the DPML process (see, HTTPBridge for more information on processing HTTP requests). [Note: 'this:param' is the local URI address space of any named arguments received on an active URI and 'param' is the name of the argument set by the HTTP bridge, hence the slightly confusing URI, this:param:param].

In the DPML process, the first instruction simply copies the parameter to a variable. If the parameter is not present then an exception will be thrown, DPML will then transfer execution to the first available exception block, which in this case has an instruction to assign a default to the variable.

The third instruction calls the DPML pipeline we met in the previous section, though in this case it calls a version which only runs the pipeline up to stage 3 (returning all speeches by Gloucester). This demonstrates how one DPML pipeline can be used to call another pipeline. As ever, DPML is simply a convenience for writing active URIs in long form, this instruction actually compiles down to...

active:dpml+operand@../pipeline2/stage3.idoc

which is the active URI to execute stage3.idoc in the DPML runtime.

The fourth instruction applies an XQuery to the speeches by Gloucester, however this time it receives both the input and the external parameter (var:param) arguments. You will see below that the XQuery uses the value of the element /nvp/search received from the external parameter as the word to find in the selected lines.

Summary

This example shows a number of things quite compactly. External parameters can be passed to any pipeline (or process), the parameters are passed on the active URI request which executes the process and are referenced within the process using the "this:param" address space.

This example also shows how pipelines can be componentized to any level of granularity and then executed, in this case with a DPML instruction which the parent DPML process issues as the URI request active:dpml+operand@../pipeline2/stage3.idoc which executes a child DPML process which reuses our previous pipeline up to, and including stage 3.

Pipeline Exception Handling

So far so good. But what if we had a problem at some point in the pipeline? For example what if the final XQuery had a syntax error like the one below ( at line 6 we've deliberately put '$speech in $input//SPEECH' when it should be 'for $speech in $input//SPEECH' ).

Try executing the process with this buggy XQuery here.

What you will see is that NetKernel detects a runtime exception and provides a detailed stack trace. The stack trace shows all of the currently executing service call stack which has the finest granularity at the bottom, and as you move up to the top progressively shows the URI request context in which the exception occurred.

The last exception shows us that a Syntax error occured, the exception above this shows that it occurred whilst attempting to compile an XQuery, the exception above this shows that it was whilst compiling "xq4-bug.xq", the exception above that shows that it occured in an "active:xquery..." request, the exception above that shows that it occured at "instr xquery with parameters input and param in DPML idoc 'interactive-bug.idoc'. etc etc all the way out to the originating request on the Jetty HTTP server.

By using the exception stack, and working upward from the finest granularity, it is usually very straightforward to locate the cause of an exception in a NetKernel pipeline.

Pipeline Debugging

The exception handling demonstration showed how you can trace problems in a badly configured or broken pipeline, what if the problem is not a runtime error but is a badly written pipeline stage producing incorrect data? We can trace these issues by inspecting the contents of a pipeline at any point by setting a break point on the pipeline.

To demonstrate, lets imagine that the result coming from the DPML stage3.idoc is not what we expected. We want to put a breakpoint between the DPML call and the final XQuery. We can do this with the NetKernel debugger.

Using the debugger set the following regular expression as a new breakpoint...

In the 'add new breakpoint' dialog set "Request URI Regex" from .* to active:dpml.*stage3.idoc.*

Now try the request for the pipeline again here, you will see that this request does not return; your browser will block waiting for a result. In another browser tab return to the debugger and inspect the 'stopped requests' - you will discover that the active:dpml request has been stopped, and that by clicking 'view' you are able to inspect the pipeline callstack, the result of stage3 and view metadata about the request and result.

When you're done, go back to 'stopped request' and click 'release', the pipeline will continue to completion and your blocked browser request will return with the result.