Guide to using Auto Escape

Introduction

Auto Escape is an optional mode of execution in the Template System developed to provide a better defense against cross-site scripting (XSS) in web applications. In this mode, the Template System assumes responsibility for applying the proper escaping modifiers for each variable in your template (and templates it may include). As such, the template developer no longer needs to manually apply escaping modifiers to each variable, a process which is repetitive and error-prone particularly in larger and more complex applications.

In order for the Template System to properly determine the escaping modifiers to apply to variables, it activates a built-in HTML-aware parser during template initialization. The parser scans the template to determines the context in which each variable is being emitted. The scanning is performed at initialization-time only hence does not contribute to processing costs at template expansion time.

Refer to Security Considerations for additional security information on escaping directives and their use according to the context in which content is being expanded.

Current Status

Auto Escape currently supports well HTML and Javascript templates. It also provides very basic support for CSS, JSON and XML templates where it applies the corresponding escaping modifier but without utilizing a parser. We may in the future provide parsers specific to these contexts and modifiers for them that are finer-grained.

Overview

Using template modifiers

In the simplest case, where you only want to use template-modifiers to achieve html-safety (that is, you only use html_escape, url_query_escape, and so forth), you can stop specifying template-modifiers in your template at all. The Auto Escape mode will generate the modifiers on its own and perform the correct escaping. You can observe the modifiers it applied to your variables using the provided command-line tool diff_tpl_auto_escape.

If you need other types of modifiers, including your own custom modifiers, you can still specify them in your template and they will continue to work just the same. The Auto Escape mode will simply apply the appropriate escaping modifiers after your own.

Other reasons for manually specifying variable modifiers include:

1. To indicate a variable is safe. You may add the :none modifier as the last modifier for a given variable, in which case the Auto Escape mode will leave it untouched and not apply escaping directives to it. Use only when you are sure the variable is trusted to be safe from XSS.
2. To select a different escaping modifier from a list of equivalent modifiers. Certain modifiers are equivalent from an escaping point of view in that they would all ensure given content is safe when escaped in a specific context. The Auto Escape mode choses the most common escaping modifier for that use but allows you to indicate a different choice as shown below:

   TemplateContext	Primary Modifier	Accepted alternatives
   TC_HTML	:html_escape	:pre_escape :html_escape_with_arg=snippet :html_escape_with_arg=attribute :url_query_escape
   TC_XML	:xml_escape	:html_escape :html_escape_with_arg=attribute

3. To add additional escaping modifiers. The Template System will never remove escaping directives you explicitly specify.

Escaping Contexts

The table belows indicates which modifier Auto-Escape selects to escape a given variable, based on the context of the variable being inserted and, possibly, on more specific information such as whether the variable is inside quotation marks. The table only applies for the well-supported TemplateContext values (currently TC_HTML and TC_JS).

In a few cases, we do not have a modifier that is both safe against XSS and respecting of the semantics of the variable therefore we fail the template initialization. An error is logged indicating the cause of the failure.

<p>Hello {{USER}}</p>
      

<img src="{{URL}}">;

<a href="/foo?q={{QUERY}}">

<form action={{URL}}>

<a href=/foo?q={{QUERY}}>My Link</a>

<div style="{{STYLE}}">

<div style={{STYLE}}>

<a href="url" onclick="doFoo('{{ARG}}');">

<a href="url" onclick="doFoo({{ARG}});">

<a href="url" onclick=doFoo('{{ARG}}');>

<b class="{{CLASS}}">

<table border={{BORDER}}>

<script>var a = '{{VALUE}}';</script>

<script>var a = {{VALUE}};</script>

Comments:

• For values of URL-accepting attributes, we apply a different modifier if the variable is at the beginning of the attribute value or not because in the former case we consider the variable to represent a complete URL and hence also check that its scheme is safe (HTTP or HTTPS).
• We recommend that you enclose attribute values in quotes in particular:
  1. Under some conditions outlined in the table above, when variables are present in attribute values that are not enclosed in quotes, we may fail template initialization because we do not have escaping modifiers that are safe to use.
  2. Also the determination of which escaping modifier to apply for unquoted attribute values specifically is likely to change in the future.

The Auto Escape mode is designed to be simple to use and to produce correct correct escaping with minimal input. It does however come with restrictions governing the use of this template system.

Limitations

• Templates may only be included in regular HTML text. Including a template within HTML comments or inside an HTML tag is currently not supported. In the example below, HEADER and FOOTER are properly included but TAG and COMMENT are not.

            <html><head>{{>HEADER}}</head><body>
            <{{>TAG}}>
            <p>
            <!-- {{>COMMENT}} -->
            <p>{{>FOOTER}}
            </body></html>
         

  If the template system detects that you are including a template in another HTML context, it will log a warning.

• Included templates may not cross HTML boundaries. They cannot start with one TemplateContext and end in another. For example, the template below crosses boundaries -- it starts in HTML and ends in Javascript -- and is therefore not a valid included template for auto-escaping.

          <html>
            <head>
            </head>
            <body>
              Hello USER.
                <script>
                   var a = 'not a nice template';
         

  To rectify this violation, continue the template by completing the javascript code and terminating it with a </script> tag.

• Error reporting. We currently report errors by logging a warning during initial template initialization and parsing if we suspect something is incorrect. Please check for them, as they indicate an error that may cause auto-escaping to work improperly.

• Beware of double escaping. When you use the Auto Escape mode, you surrender all variable escaping to the template system. If you also perform your own escaping in your source code, you may face situations where content was escaped multiple times.

  In particular, do not use any of the legacy methods below or others that perform variable escaping.

  • SetEscapedValue
  • SetEscapedFormatttedValue
  • SetEscapedValueAndShowSection

• Supported HTML contexts. HTML (TC_HTML) and Javascript (TC_JS) templates are well supported. Other template contexts have only basic support. For these contexts, variables are escaped as follows:

  TemplateContext	Escaping Applied
  TC_JSON	:javascript_escape
  TC_XML	:xml_escape
  TC_CSS	Same escaping as for TC_HTML

• New restrictions on the use of BI_SPACE and BI_NEWLINE. These system variables may not be re-assigned to a semantically different value via dict->SetValue or related methods, and they may not have modifiers attached to them in templates. For instance: it's ok to redefine BI_NEWLINE from \n to \r\n, since html treats them identically. But changing it to hello, world! is not allowed.

A new filename convention

The Auto Escape feature codifies simple conventions of template filenames:

1. CSS templates optionally include, in their filename, the keywords style or stylesheet or css.
2. Javascript templates optionally include, in their filename, the keywords js or javascript.

If we find one of these keywords in the filename but it does not match the template type given, we log a warning.

Development Tools

diff_tpl_auto_escape: Escaping Modifier Comparaison tool for Templates

If you currently apply escaping modifiers to variables in your template, you may be interested in diff_tpl_auto_escape. This tool compares escaping modifiers applied to each variable in a template against the modifiers that Auto Escape would have selected and emits information on a mismatch. Cases where it may be helpful include:

1. You are considering switching to Auto-Escape and would like to know whether your variables will be escaped differently.
2. You want to continue manually escaping all variables but would like some validation that your escaping is correct. Consider using the --dump_templates command-line option described below.

Consider the example below of an HTML template with two variables, USER which is escaped properly and ID which is not.

<html>
  <body>
    <p>Hello {{USER:h}}</p>
    <script>var id = '{{ID:h}}'; alert(id);</script>
  </body>
</html>

Running diff_tpl_auto_escape on this template will emit one line for variable ID indicating the mismatch and how the escaping you provided would be modified under Auto-Escape. Note that by default, the tool will not output information for variables that do not have escaping modifiers applied to them in your template. In that case you may want to provide the --dump_templates command-line option.

Important command line flags:

• --context -- sets the TemplateContext for the template files provided. Default: "TC_HTML"
• --dump_templates -- emits the entire template showing the modifiers it selected to each variable.
• --template_dir -- sets the template root directory. Default: ./ which is the correct specification when it is run from the directory where the templates are located. This is only used if the input template filenames are specified as relative paths rather than absolute paths.

For a full list of command line flags, run diff_tpl_auto_escape --help.

• Howto
• Tips
• Example