SLaks.Blog

Immutability, part 1: Read-only vs. Immutable

Tue, 11 Jun 2013 00:00:00 -0700

A read-only object is an object that does not expose any way to change it. ReadOnlyCollection<T> (returned by AsReadOnly()) is a good example). However, IEnumerable<T> is also read-only.
The important distinction is that read-only objects are allowed to change.

If you write list.Add(new Test()), your read-only collection (which just wraps list) will have changed.

Read-only collections are useful for designing safe APIs, where only the owner of the collection is allowed to change it.
However, it won't do any good for thread-safety.

An immutable object is an object that cannot change at all, no matter what happens (Reflection doesn't count). string is an excellent example of an immutable class; the value of an existing string instance can never change, no matter what happens. (barring reflection, unsafe code, and certain marshalling tricks)

.Net does not have any built-in immutable collections, but the CLR team released a library of immutable collection types on NuGet.

Truly immutable objects are intrinsically thread-safe. Since there is no way to modify them, there is no chance that other threads will observe an inconsistent instance.

In summary, there are four kinds of “unchangeable” things:

readonly (ReadOnly in VB.Net; final in Java) fields.
These fields cannot be re-assigned to point to a different instance. However, nothing prevents you from mutating the instance pointed to by the field.
Read-only classes
These classes do not expose any APIs to mutate their contents, but can change by other means (typically, by changes made directly to the mutable objects they wrap, or in response to events handled within the class).
Examples include ReadOnlyCollection<T>.
API-enforced immutable classes
These classes are not intrinsically immutable, but have an API design that guarantees that mutations will not happen. In other words, they may contain mutable state, but they will never actually mutate that state. Examples include delegates (MulticastDelegate has an array of handlers, which it will never mutate).
Compiler-enforced immutable classes
These classes only contain readonly fields that are themselves immutable. This means that the compiler will report an error if the developer mistakenly tries to mutate it. This provides an additional layer of defence against potential mistakes.
Examples include most EventArgs classes.

Finally, an object is deeply immutable (or read-only) if both it and all of its properties are also deeply immutable (or read-only). In other words, the entire graph of objects reachable from it must be immutable (or read-only).

Next time: Creating a simple immutable stack

Writing the endraw tag in Jekyll code blocks

Mon, 10 Jun 2013 00:00:00 -0700

Last time, we saw how to write about Jekyll tags in Jekyll-based blog posts, using HTML entities or the {% raw %} block.

These techniques cannot be used in syntax-highlighted code blocks (Jekyll's {% higlight %} tag or a Markdown code block), since such blocks are always HTML-escaped. Instead, you can wrap all of the code in the block with a Liquid {% raw %} tag. Since the Liquid tag is processed before Markdown or syntax highlighting, this works perfectly.

{% raw %}
Liquid uses tags like {% if %} or {% for %}.
It also supports variable interpolation: {{ someVariable }}
{% endraw %}

This approach works wonderfully, until you try to put the {% endraw %} tag in a code block. You can't put that in a {% raw %} block, because it will close the block. You can't use entities, because text within highlighted code blocks is always HTML-escaped.

One potential option would be to break apart the tag; something like

{% raw %}
This is how you show the termination of the `{% raw %}` tag inside itself: 
{% {% endraw %}endraw %}{% raw %}

However, due to a bug in Liquid, this doesn't work correctly. This markup is supposed to mean the following:

Write the literal text {% within the {% raw %} block
Terminate the {% raw %} block with {% endraw %}
Write the rest of the of the tag (endraw %})
Re-enter the {% raw %} block to continue with other markup

What actually happens is that Liquid ignores the actual {% endraw %} command, and treats the entire line as raw text. In other words, the code you see in the above example is (incorrectly) rendered exactly as written. The literal text {% before the {% endraw %} causes Liquid to ignore the tag completely, breaking this technique.

To work around this bug, we need to put the {% text outside the {% raw %} block. However, Liquid does not have any way to escape a {, and we can't use HTML escaping inside the highlighted code block, so there is no obvious way to write the {% without breaking the parser.

Variables can help here. We can create a Liquid variable that holds the literal text {%, then interpolate this variable outside the {% raw %} block. For example:

{% assign openTag = '{%' %}
{% raw %}
This is how you show the termination of the `{% raw %}` tag inside itself: 
{% endraw %}{{ openTag }} endraw %}{% raw %}
This content is back inside the {% raw %} block
{% endraw %}

This code is parsed like this:

Terminate the {% raw %} block with {% endraw %}
Write the value of the openTag variable ({%) with {{ openTag }}
Write the remainder of the {% endraw %} tag as literal text with endraw %}
Re-enter the {% raw %} block for additional raw content

This technique can also be used to write tags in inline code: `{{ openTag }} sometag %}`

If these worksarounds seem completicated, writing this post itself (also in Liquid Markdown!) was even more complicated.
See the source to see how I did it.

Writing about Jekyll in Jekyll

Sun, 09 Jun 2013 00:00:00 -0700

Jekyll is a very nice system for writing blogs. However, it does have some shortcomings, particularly in the Liquid templating engine. In this post, I will talk about how to write about Liquid tags within a Liquid file (such as a Jekyll blog post). The problem is that writing Liquid syntax such as tags or variables in the content will cause Liquid to interpret them as commands, and break evertything.

This problem can occur when writing a blog post about Liquid itself (such as this one), or when writing a blog post about code that generates Liquid markup (like I did earlier).

You may want to mention Liquid syntax in two ways: inline code (mentioning a {% tag %} in a sentence), and multi-line syntax highlighted code blocks (including a whole chunk of Liquid markup in a post).

Liquid includes a tag specifically designed to solve this problem: the {% raw %} tag. For example, the Liquid Markdown used to produce the preceding sentence is:

Liquid includes a tag specifically designed to solve this problem: 
the `{% raw %}{% raw %}{% endraw %}` tag.

I wrap the text I'm trying to produce ({% raw %}) in {% raw %}...{% endraw %} tags to prevent liquid from treating the content itself as a tag. Since Liquid processes the text before the Markdown parser sees it, the resulting Markdown source is simply `{% raw %}`, which is exactly what I want.

All that overhead is very annoying when writing a simple tag. We can make it simpler by writing part of the tag as an HTML entity, so that Liquid doesn't recognize it as a tag: {% tag %}. The browser will display the HTML entity as a regular {, but Liquid won't recognize it. However, this does mean that we can't use Markdown ` blocks to create the code block, since that will escape the HTML and make it display a literal {.
Thus, the final approach is <code>{% tag %}</code>. Although this isn't much shorter than the Liquid raw tag, I find it more readable, since it doesn't nest Liquid content within Liquid commands.

Next time: How can you do this inside a syntax-highlighted code block?

Migrating client-side syntax highlighting to Jekyll

Sun, 02 Jun 2013 00:00:00 -0700

The next step in migrating my blog to Jekyll was to convert the code blocks to use Jekyll's {% highlight %} tag.

Since Blogger has no support for server-side syntax highlighting, all of the code blocks in my HTML are implemented as <pre class="brush: someLanguage">...</pre>, with HTML-escaped code inside the tag. (the class name is used by SyntaxHighlighter) I needed to convert that to Liquid tags with raw (non-escaped) code inside of them.

To do this, I wrote a small C# script:

const string PostsFolder = @".../_posts";
var langMappings = new Dictionary<string, string>{
    { "vb", "vb.net" }
};
Func<string, string> GetLanguage = lang => {
    string mapped;
    if (langMappings.TryGetValue(lang, out mapped))
        return mapped;
    return lang;
};

Func<string, MatchEvaluator> Replace = replacement => ManifestResourceInfo => ManifestResourceInfo.Result(replacement);

var regexes = new [] {
    Tuple.Create<Regex, MatchEvaluator>(
        new Regex(@"\s*\<pre\s*class=""brush:\s*(\w+);?\s*"">\n*(.*?)\n*</pre>\s*", RegexOptions.Singleline),
        m => "\n{% endraw %}\n{% highlight " + GetLanguage(m.Groups[1].Value) +" %}\n" 
            + WebUtility.HtmlDecode(m.Groups[2].Value) 
            + "\n{% endhighlight %}\n{% raw %}\n"
    ),
    Tuple.Create(new Regex(@"{% raw %}\s*{% endraw %}\s*", RegexOptions.Singleline), Replace("")),
    Tuple.Create(new Regex(@"<font face=""Courier New"">([^<]+)</font>", RegexOptions.Singleline), Replace("<code>$1</code>"))
};

foreach (var file in Directory.EnumerateFiles(PostsFolder, "*.html")) {
    File.WriteAllText(file,
        regexes.Aggregate (File.ReadAllText(file), (txt, tuple) => tuple.Item1.Replace(txt, tuple.Item2))
    );
}

(This kind of script is most easily run in LINQPad or scriptcs)

This code loops through every HTML file in PostsFolder and runs the text through a series of regular expressions (yes, evil) to update it for Jekyll.

The first, and biggest, regex matches <pre class="brush: someLanguage">...</pre>, and convers each to Jekyll {% highlight someLanguage %} blocks. Since SyntaxHighlighter uses different language names than Pygments', I have a langMappings that maps language names from the HTML to language names for the Jekyll output. I also un-HTML-escape the contents of each code block. Finally, because I modified blogger2jekyll to wrap each post in a {% raw %} tag, it terminates the {% raw %} and re-enters it after the code.

Depending on what your code blocks look like, you may want to change it to wrap the contents of each {% highligh %} tag in a {% raw %} block too.

The next regex strips empty {% raw %} blocks in case there are two code blocks in a row (which I had here).

The last regex replaces bad non-semantic <font> tags created by Windows Live Writer with <code> tags.
If your imported HTML has similar issues, you can add more regexes to correct them.

Next time: Preserving the RSS feed

Migrating from Blogger to Jekyll

Fri, 31 May 2013 00:00:00 -0700

The first step in my migration to Jekyll was to import my old posts into the Jekyll site. To do this, I used blogger2jekyll, a wonderful open-source Node.js script that does exactly that.

Using this tool is very simple. First, log into Blogger's admin panel, got to Settings, Other, and click Export blog to download a giant XML file with all of your posts.

Next, install and run the script: (you'll need to install Node.js first)

npm install -g blogger2jekyll
blogger2jekyll  /path/to/blog-dd-mm-yyyy.xml ./_posts

If you aren't running it from the directory containing your Jekyll site, you'll need to specify the full path to Jekyll's _posts directory.

This script will create HTML files with the contents of each post from the exported blog, ready for Jekyll to serve. It will include layout: "post" in the Jekyll front matter; if you have a different layout name, you'll need to do a bulk replace within the resulting files. It will also set the permalink for each post so that existing posts keep their old URLs (even if the Jekyll blog hasa different URL scheme).

I had a couple of problems with this script:

When running on Windows, the generated permalink URLs use backslashes rather than forward slashes
Posts that contain Liquid-like markup will break Jekyll with a parse error
The script also imports comments directly into the resulting HTML, which is rarely a good idea.

I fixed all of these issues in a pull request, which has been merged and pushed to npm.

I changed it to wrap the contents of each post in a Liquid {% raw %} tag. I also added an internal option to skip comments. However, I didn't add a command-line interface for the comment option; to use it, you'll need to manually add , skipComments: true to the parse() call.

Next time: Converting Code Blocks

About the new design

Mon, 27 May 2013 00:00:00 -0700

My new design is powered by Jekyll and LESS (the LESS does as much or more as the Jekyll).

When implementing the design, I had the following goals in mind:

No costs
- I use GitHub Pages for completely free hosting (other than the cost of the domain name)
- This means that I cannot use Octopress or Jekyll plugins
No build step
- I want to be able to edit posts from anywhere, without having to install Ruby or Grunt.js and run any kind of build process before pushing
- I do use pre-compiled LESS, since Jekyll on GitHub Pages cannot compile LESS, and I really want to use LESS. (also, since LESS needs to be edited far less often than post content, and since my editor automatically compiles LESS files on save
No Javascript
- Especially with the power of CSS selectors, there should be no reason to use Javascript for static content
- If I add comments, I will have to relax this restriction (since the site will no longer be purely static content)
DRY implementation
- Repetition of data or rules is a problem not only in large programs, but also in HTML templating engines or style definitions. LESS in particular is very helpful in getting rid of unnecessary repetition in CSS.
Rich category support
- I feel that category listing pages are a great way to make posts more discoverable
Clean, minimalist, design
- One of the mistakes of my old design was that it put too little focus on content (both in size and in coloring).
Easy navigation within series of multiple posts
- Coming soon
- For now, the new sidebar in post pages (see left side) provides this to a limited extent
Excessive use of bulleted lists
- Bulleted lists are an excellent way to concisely present structured data

Next time: How I migrated from Blogger to Jekyll

Relaunch!

Sun, 26 May 2013 00:00:00 -0700

After nearly a year of inactivity, I have finally returned to my blog.

I had neglected it for so long primarily because I don't like Blogger's compose options. After spending so much time on StackOverflow and GitHub, I find Markdown far more convenient than Windows Live Writer or Blogger's compose window, especially when writing about code. I also was never very happy with the design I originally created, and doing raw HTML / CSS design in Blogger is painful.

To solve these problems, I just finished porting my blog to Jekyll on GitHub Pages. Now that I can write posts in my favorite Markdown editor, I hope to end up writing much more.

In the next few posts, I'll write about how I designed the new blog and how I migrated my existing content.

My old blog is still up at old-blog.slaks.net. However, all of the content has been migrated to the new one (with the same URLs), so that should not be necessary. If you see anything wrong with the new site, let me know on Twitter or by email.

Visual Studio 2012 and webpages:Version

Mon, 11 Jun 2012 22:33:00 -0700

If you open an older ASP.Net MVC3 project in Visual Studio 2012, you may see lots of errors in the Razor views, along the lines of “The name 'model' does not exist in the current context”, and similar errors whenever you try to use MVC features like HTML helpers or ViewContext (eg, “System.Web.WebPages.Html.HtmlHelper does not contain a definition for TextBoxFor”).

This happens if there is no <add key="webpages:Version" value="1.0" /> in the <appSettings> element in Web.config.

Without this element, Visual Studio will assume that you’re using the latest version of Razor and the WebPages framework. Until VS2012, this wasn’t a problem, since there was only one version. However, since VS2012 ships with ASP.Net WebPages 2.0, the IDE will load this version by default. Since you’ve specified the MVC integration & <configSection> for 1.0 (in Views/Web.config, since all of the assembly references specify Version=1.0.0.0), the language services won’t load the MVC settings. Therefore, the @model directive will not work, and the view will inherit the standard WebPage base class rather than the MVC WebViewPage base class (which contains the MVC HTML helpers)

This issue has no effect at runtime because the server won’t load any version 2.0 assemblies.

Exploring Caller Info Attributes

Thu, 01 Mar 2012 06:57:00 -0800

Last year, Microsoft announced a simple new feature in C# 5: Caller Info Attributes. These attributes let you to create methods with optional parameters and tell the compiler to pass the caller’s filepath, line number, or member name instead of the parameter’s default value. This allows you to create logging methods that automatically know where they’re being called.

When the feature was announced, I wrote a couple of blog posts that delved into some of the corner cases of the new feature. At the time, there was no public implementation, so they were pure conjecture.

This morning, Microsoft released the beta of Visual Studio 11, which is the first public build supporting these attributes. Now, I can finally test my theories. Here are the results:

Although these classes are new to the .Net Framework 4.5, you can still use this feature against older framework versions by creating your own classes in the System.Runtime.CompilerServices namespace. However, the feature will only work if the code calling the method is compiled with the C# 5 compiler; older compilers will ignore the attributes and simply pass the parameters’ default values.

All of the attributes can only be applied to arguments of types that have standard (not custom) implicit conversions to int or string. This means that it isn’t practical to overflow [CallerLineNumber] (the compiler ran out of memory first), so I can’t test how that behaves.

Using [CallerMemberName] on field initializers passes the field name, and on static or instances constructors passes the string ".cctor" or ".ctor" (as documented) In indexers, it passes "Item".

If a class has a constructor that takes only caller info attribute parameters, and you create another class that inherits it and does not declare a constructor (thus implicitly passing optional parameters), it passes the line number and file name of the class keyword in the derived class, but leaves the declared default for the member name (I suspect that’s a bug).

If you do declare a constructor, it passes the string ".ctor" as the member name for the implicit base() call (just like a normal method call from inside a constructor) and the line number of the beginning of the constructor declaration. If you actually write a base() call, it passes the line number of the base keyword.

If a call spans multiple lines, [CallerLineNumber] passes the line containing the openning parenthesis.

Delegates are fully supported; if you call a delegate that has an argumented annotated with a caller info attribute, the compiler will insert the correct value, regardless of the method you’re actually calling (which the compiler doesn’t even know).

LINQ query comprehension syntax is not supported at all; if you create a (for example) Select() method that contains a caller info attribute, then call it from a LINQ query (not lambda syntax), the compiler will crash (!). (they will fix that)

Expression trees do not support optional parameters at all, so that corner case is irrelevant.

Attributes are the most interesting story. What should happen if you declare a custom attribute that takes parameters with caller info attributes, then apply that attribute in various cases? This could potentially be very useful, since there is currently no way for an attribute to know what it’s being applied to. (I hadn’t thought of this usage when I wrote the original blog post)

The documentation says that this will work in all cases, and that [CallerMemberName] will pass whatever the attribute is being applied to. However, in the beta build, this doesn’t always work.

Attributes applied to method arguments or return values do not pass any caller info at all. Attributes applied to types or generic type arguments do not pass member names (this is very disappointing)

Hopefully, those will be fixed before release.

ASP.Net MVC Unobtrusive Validation Bug

Tue, 21 Feb 2012 18:15:00 -0800

If you use the ASP.Net MVC 3 [Compare] validation attribute on a model property, then include that model as a property in a parent model (so that the field name becomes Parent.ChildProperty), the built-in unobtrusive client validation will choke, and will always report the field as having an error.

This is due to a bug on line 288 of jquery.validate.unobtrusive.js:

adapters.add("equalto", ["other"], function (options) {
    var prefix = getModelPrefix(options.element.name),
        other = options.params.other,
        fullOtherName = appendModelPrefix(other, prefix),
        element = $(options.form).find(":input[name=" + fullOtherName + "]")[0];	// Bug

    setValidationValues(options, "equalTo", element);
});

Because the value of the name attribute selector is not quoted, this fails if the name contains a ..

The simplest fix is to add quotes around the concatenated value. However, the jQuery selector there is overkill. HTML form elements have properties for each named input, so you can do this instead:

adapters.add("equalto", ["other"], function (options) {
    var prefix = getModelPrefix(options.element.name),
        other = options.params.other,
        fullOtherName = appendModelPrefix(other, prefix),
        element = options.form[fullOtherName];
    if (!element)
        throw new Error(fullOtherName + " not found");
    //If there are multiple inputs with that name, get the first one
    if (element.length && element[0])        
        element = element[0];
    setValidationValues(options, "equalTo", element);
});

Protecting against CSRF attacks in ASP.Net MVC

Wed, 25 Jan 2012 23:18:00 -0800

CSRF attacks are one of the many security issues that web developers must defend against. Fortunately, ASP.Net MVC makes it easy to defend against CSRF attacks. Simply slap on [ValidateAntiForgeryToken] to every POST action and include @Html.AntiForgeryToken() in every form, and your forms will be secure against CSRF.

However, it is easy to forget to apply [ValidateAntiForgeryToken] to every action. To prevent such mistakes, you can create a unit test that loops through all of your controller actions and makes sure that every [HttpPost] action also has [ValidateAntiForgeryToken].

Since there may be some POST actions that should not be protected against CSRF, you’ll probably also want a marker attribute to tell the test to ignore some actions.

This can be implemented like this:

First, define the marker attribute in the MVC web project. This attribute can be applied to a single action, or to a controller to allow every action in the controller.

///<summary>Indicates that an action or controller deliberately 
/// allows CSRF attacks.</summary>
///<remarks>All [HttpPost] actions must have 
/// [ValidateAntiForgeryToken]; any deliberately unprotected 
/// actions must be marked with this attribute.
/// This rule is enforced by a unit test.</remarks>
[AttributeUsage(AttributeTargets.Class | AttributeTargets.Method)]
public sealed class AllowCsrfAttacksAttribute : Attribute { }

Then, add the following unit test:

[TestMethod]
public void CheckForCsrfProtection() {
    var controllers = typeof(MvcApplication).Assembly.GetTypes().Where(typeof(IController).IsAssignableFrom);
    foreach (var type in controllers.Where(t => !t.IsDefined(typeof(AllowCsrfAttacksAttribute), true))) {
        var postActions = type.GetMethods()
                                .Where(m => !m.ContainsGenericParameters)
                                .Where(m => !m.IsDefined(typeof(ChildActionOnlyAttribute), true))
                                .Where(m => !m.IsDefined(typeof(NonActionAttribute), true))
                                .Where(m => !m.GetParameters().Any(p => p.IsOut || p.ParameterType.IsByRef))
                                .Where(m => m.IsDefined(typeof(HttpPostAttribute), true));

        foreach (var action in postActions) {
            //CSRF XOR AntiForgery
            Assert.IsTrue(action.IsDefined(typeof(AllowCsrfAttacksAttribute), true) != action.IsDefined(typeof(ValidateAntiForgeryTokenAttribute), true),
                            action.Name + " is [HttpPost] but not [ValidateAntiForgeryToken]");
        }
    }
}

typeof(MvcApplication) must be any type in the assembly that contains your controllers. If your controllers are defined in multiple assemblies, you’ll need to include those assemblies too.

The Dark Side of Covariance

Thu, 15 Dec 2011 04:14:00 -0800

What’s wrong with the following code?

var names = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
...
if (names.Contains(sqlCommand.ExecuteScalar())

This code is intended to check whether the result of a SQL query is contained in a case-insensitive collection of names. However, if you run this code, the resulting check will be case-sensitive. Why?

As you may have guessed from the title, this is caused by covariance. In fact, this code will not compile at all against .Net 3.5.

The problem is that ExecuteScalar() returns object, not string. Therefore, it doesn’t call HashSet<string>.Contains(string), which is what it’s intending to call (and which uses the HashSet’s comparer). Instead, on .Net 4.0, this calls the Enumerable.Contains<object>(IEnumerable<object>, string) extension method, using the covariant conversion from IEnumerable<string> to IEnumerable<object>. Covariance allows us to pass object to the Contains method of any strongly-typed collection (of reference types).

Still, why is it case-sensitive? As Jon Skeet points out, the LINQ Contains() method is supposed to call any built-in Contains() method from ICollection<T>, so it should still use the HashSet’s case-insensitive Contains().

The reason is that although HashSet<String> implements ICollection<string>, it does not implement ICollection<object>. Since we’re calling Enumerable.Contains<object>, it checks whether the sequence implements ICollection<object>, which it doesn’t. (ICollection<T> is not covariant, since it allows write access)

Fortunately, there’s a simple fix: just cast the return value back to string (and add a comment explaining the problem). This allows the compiler to call HashSet<string>.Contains(string), as was originally intended.

//Call HashSet<string>.Contains(string), not the
//covariant Enumerable.Contains(IEnumerable<object>, object)
//http://blog.slaks.net/2011/12/dark-side-of-covariance.html
if (names.Contains((string)sqlCommand.ExecuteScalar())

(I discovered this issue in my StringListConstraint for ASP.Net MVC)

CAPTCHAs do not mitigate XSS worms

Thu, 08 Dec 2011 02:32:00 -0800

One common misconception about web security is that protecting important actions with CAPTCHAs can prevent XSS attacks from doing real damage. By preventing malicious code from scripting critical tasks, the idea goes, XSS injections won’t be able to accomplish much.

This idea is dangerously wrong.

First of all, this should not even be considered except as a defense-in-depth mechanism. Regardless of whether the actions you care about are protected by CAPTCHAs, XSS attacks can create arbitrary UI on your pages, and can thus make “perfect” phishing attacks.

Also, even with CAPTCHAs, an XSS injection can wait until the user performs the critical action, then change the submitted data to the attacker’s whim.

For example, if Twitter took this approach to prevent XSS injections from sending spammy tweets, the attacker could simply wait until the user sends a real tweet, then silently append advertising to the tweet as the user submits it and fills out the CAPTCHA.

However, there is also a more fundamental issue. Since the injected Javascript is running in the user’s browser, it simply display the CAPTCHA to the user and block all page functionality until the user solves the CAPTCHA. The attacker can even put his own text around the CAPTCHA to look like a legitimate security precaution, so that the (typical) user will not realize that the site has been compromised. (that could be prevented by integrating a description of the action being performed into the CAPTCHA itself in a way that the attacker can’t hide)

I haven’t even mentioned the inconvenience of forcing all legitimate, uncompromised users to fill out CAPTCHAs every time they do anything significant.

In summary, CAPTCHAs should only be used to prevent programs from automatically performing actions (eg, bulk-registering Google accounts), and as a rate-limiter if a user sends too many requests too quickly (eg, getting a password wrong too many times in a row).

XSS can only be stopped by properly encoding all user-generated content that gets concatenated into markup (whether HTML, Javascript, or CSS)

About Concurrent Collections

Sun, 20 Nov 2011 18:42:00 -0800

One of the most useful additions to the .Net 4.0 base class library is the System.Collections.Concurrent namespace, which contains an all-new set of lock-free thread.

However, these collections are noticeably different from their classical counterparts. There is no simple ConcurrentList<T> that you can drop into your code so that it will become thread-safe. Instead, the new namespace has a queue, a stack, and some new thing called a bag, as well as ConcurrentDictionary<TKey, TValue> that largely resembles classical dictionaries. It also has a BlockingCollection<T> class that wraps a concurrent collection and blocks until operations can succeed.

Many people have complained that Microsoft chose to provide an entirely new set of classes, rather than adding synchronized versions of the existing collections.

In truth, however, creating this new set of classes is the correct – and, in fact, only – choice. Ordinary synchronized are rarely useful and will not make a program thread-safe. In general, slapping locks everywhere does not make a program thread-safe! Rather, that will either not help (if there aren’t enough locks) or, if there are enough locks, result in deadlocks or a program that never runs more than one thread at a time. (depending on how many different objects get locked)

Collections in particular have two fundamental issues when used on multiple threads.

The first, and simpler, issue is that the collection classes themselves are not thread-safe. If two threads add to a List<T> at the same exact time, one thread is likely to overwrite the other thread’s value. A synchronized version of List<T> with a lock around every method would solve this problem.

The bigger issue is that any code that uses a List<T> is unlikely to be thread-safe, even if the list itself is thread-safe. For example. you can never enumerate over a multi-threaded list, because another thread may change the list at any time, invalidating the enumerator. This issue could be solved by taking a read lock for the lifetime of the enumerator. However, that is also a bad idea, since if any client code forgets to dispose the enumerator, the collection will deadlock when written to.

You also cannot use indices. It is never safe to get, set, or remove an item at an index, because another thread might remove that item or clear the entire collection between your index check and the operation.

To solve all of these problems, you need thread-safe collections that provide atomic operations. This is why all of the concurrent collections have such strange methods, including TryPop, AddOrUpdate, and TryTake. These methods perform their operations atomically, and return false if the collection was empty (as appropriate; consult the documentation for actual detail). Thus, the new concurrent collections can be used reliably in actual multi-threaded code without a separate layer of locks.

Beware of Response.RedirectToRoute in MVC 3.0

Wed, 09 Nov 2011 04:00:00 -0800

ASP.Net MVC uses the new (to ASP.Net 3.5) Http*Base wrapper classes (HttpContextBase, HttpRequestBase, HttpResponseBase, etc) instead of the original Http* classes. This allows you to create mock implementations that inherit the Http*Base classes without an actual HTTP request. This is useful for unit testing, and for overriding standard behaviors (such as route checking).

In ordinary MVC code, the HttpContext, Request, and Response properties will return Http*Wrapper instances that directly wrap the original Http* classes (eg, HttpContextWrapper, which wraps HttpContext). Most MVC developers use the HttpContext and related properties without being aware of any of this redirection.

Until you call Response.RedirectToRoute. This method, which is new to .Net 4.0, redirects the browser to a URL for a route in the new ASP.Net routing engine. Like other HttpResponse methods, HttpResponseBase has its own version of this method for derived classes to override.

However, in .Net 4.0, Microsoft forgot to override this method in the standard HttpResponseWrapper. Therefore, if you call Response.RedirectToRoute in an MVC application (where Response is actually an HttpResponseWrapper), you’ll get a NotImplementedException.

You can see this oversight in the methods list for HttpResponseWrapper. Every method except for RedirectToRoute and RedirectToRoutePermanent are list as (Overrides HttpResponseBase.MethodName().); these methods are listed as (Inherited from HttpResponseBase.MethodName().)

To work around this issue, you can either use the original HttpResponse by writing HttpContext.Current.Response.RedirectToRoute(…) or by calling Response.Redirect instead.

Note that most MVC applications should not call Response.Redirect or Response.RedirectToRoute at all; instead, they should return ActionResults by calling helper methods like return Redirect(…); or return RedirectToAction(…);

In the upcoming ASP.Net 4.5 release, these methods have been properly overridden.

Caller Info Attributes vs. Stack Walking

Sun, 23 Oct 2011 23:55:00 -0700

People sometimes wonder why C# 5 needs to add caller info attributes, when this information is already available by using the StackTrace class. In reality, caller info attributes behave rather differently from the StackTrace class, for a number of reasons.

Advantages to Caller Info Attributes

The primary reason to use caller info attributes is that they’re much faster. Stack walking is one of the slowest internal (as opposed to network IO) things you can do in .Net (disclaimer: I haven’t measured). By contrast, caller info attributes have exactly 0 performance penalty. Caller info is resolved at compile-time; the callsite is compiled to pass a string or int literal that was determined by the compiler. Incidentally, this is why C# 5 doesn’t have [CallerType] or [CallerMemberInfo] attributes; the compiler team wasn’t happy with the performance of the result IL and didn’t have time to implement proper caching to make it faster.

Caller info attributes are also more reliable than stack walking. Stack walking will give incorrect results if the JITter decides to inline either your method or its caller. Since caller info attributes are resolved at compile-time, they are immune to inlining. Also, stack walking can only give line number and filename info if the calling assembly’s PDB file is present at runtime, whereas [CallerFileName] and [CallerLineNumber] will always work.

Caller info attributes are also unaffected by compiler transformations. If you do a stack walk from inside a lambda expression, iterator, or async method, you’ll see the compiler-generated method; there is no good way to retrieve the name of the original method. Since caller info attributes are processed by the compiler, [CallerMemberName] should correctly pass the name of the containing method (although I cannot verify this). Note that stack walking will retrieve the correct line number even inside lambda expressions (assuming there is a PDB file)

Advantages To Stack Walking

There are still a couple of reasons to use the StackTrace class. Obviously, if you want to see more than one frame (if you want your caller’s caller), you need to use StackTrace. Also, if you want to support clients in languages that don’t feature caller info attributes, such as C# 4 or F#, you should walk the stack instead.

Stack walking is the only way to find the caller’s type or assembly (or a MemberInfo object), since caller info attributes do not provide this information.

Stack walking is also the only choice for security-related scenarios. It should be obvious that caller info attributes can trivially be spoofed; nothing prevents the caller from passing arbitrary values as the optional parameter. Stack walking, by contrast, happens within the CLR and cannot be spoofed.

Note that there are some other concerns with stack walking for security purposes.

Filename and line number information can be spoofed by providing a modified PDB file
Class and function names are meaningless; your enemy can name his function whatever he wants to. All you should rely on is the assembly name.
There are various ways for an attacker to cause your code to be called indirectly (eg, DynamicInvoking a delegate) so that his assembly isn’t the direct caller. In fact, if the attacker invokes your delegate on a UI thread (by calling BeginInvoke), his assembly won’t show up on the callstack at all. The attacker can also compile a dynamic assembly that calls your function and call into that assembly.
As always, beware of inlining

Subtleties of C# 5’s new [CallerMemberName]

Tue, 18 Oct 2011 00:31:00 -0700

UPDATE: Now that the Visual Studio 11 beta has shipped with this feature implemented, I wrote a separate blog post exploring how it actually behaves in these corner cases.

Last time, I explored various pathological code samples in which the [CallerLineNumber] attribute does not have obvious behavior. This time, I’ll cover the last of these new caller info attributes: [CallerMemberName].

The [CallerMemberName] attribute tells the compiler to insert the name of the containing member instead of a parameter’s default value. Unlike [CallerLineNumber] and [CallerFileName], this has no equivalent in C++; since the C / C++ versions of these features are in the preprocessor, they cannot be aware of member names.

Most calls to methods with optional parameters take place within a named method, so the behavior of this attribute is usually obvious. However, there are a couple of places where the exact method name is not so obvious.

If you call a [CallerMemberName] method inside a property or event accessor, what name should the compiler pass? Common sense indicates that it should pass the name of the property or event, since that’s the name you actually see in source code. That would also allow this attribute to be used for raising PropertyChanged events. However, this option doesn’t pass enough information, since it would not be possible to determine whether it was called from the getter or the setter. To expose the maximal amount of information, the compiler should pass the name of the actual method for the accessor – get_Name or set_Name.

I would assume that the compiler only passes the property name, since that is what most people would probably expect.

A less-trivial question arises when such a method is called from a constructor or static constructor. Should the compiler just pass the name of the class, since that’s what the member is named in source code? If so, there would be no way to distinguish between an instance constructor and a static constructor. Should the compiler pass the actual names of the CLR methods (.ctor and .cctor)? If so, there would be no way to tell the class name, which is worse. Should it pass both (ClassName.ctor)? That would expose the maximal amount of information, but wouldn’t match the behavior in other members, which does not include the class name.

On a related note, what about calls to base or this constructors that take [CallerMemberName] arguments? Is that considered part of the class’ constructor, even though the call is lexically scoped outside the constructor? If not, what should it pass?

A further related concern is field initializers. Since field initializers aren’t explicitly in any member, what should the compiler pass if you call a [CallerMemberName] method in a field initializer? I would assume that they’re treated like contructors (or static constructors for static field initializers)

I would assume that a call to a [CallerMemberName] method from within an anonymous method or LINQ query would use the name of the parent method.

The most interesting question concerns attributes. What should happen if you declare your own custom attribute that takes a [CallerMemberName] parameter, then apply the attribute somewhere without specifying the parameter?

If you place the attribute on a parameter, return value, or method, it would make sense for the compiler to pass the name of the method that the attribute was applied to. If you apply the attribute to a type, it might make sense to pass the name of that type. However, there is no obvious choice for attributes applied to a module or assembly.

I suspect that they instead chose to not pass these caller info in default parameters for attribute declarations, and to instead pass the parameters’ declared default values. If so, it would make sense to disallow caller info attributes in attribute constructor parameters. However, this would also prevent them from being used for attributes that are explicitly instantiated in normal code (eg, for global filters in MVC).

Next Time: Caller Info Attributes vs. Stack Walking

Subtleties of C# 5’s new [CallerLineNumber]

Fri, 07 Oct 2011 15:20:00 -0700

UPDATE: Now that the Visual Studio 11 beta has shipped with this feature implemented, I wrote a separate blog post exploring how it actually behaves in these corner cases.

This is part 2 in a series about C# 5’s new caller info attributes; see the introduction.

The [CallerLineNumber] attribute tells the compiler to use the line number of the call site instead of the parameter’s default value. This attribute has more corner cases than [CallerFileName]. In particular, unlike the C preprocessor’s __LINE__ macro, the C# compiler inserts the line number of a parsed method call. Therefore, it is not always clear which line a method call expression maps to.

What should this call print:

static class Utils {
    int GetLine([CallerLineNumber] int line = 0) {
        return line;
    }
}

Console.WriteLine(
    Utils
        .
        GetLine
        (
        )
    );

Should it print the line number that the statement started? The line in which the call to GetLine started? The line containing the parentheses for GetLine? What if it’s in a multi-line lambda expression?

There are also a few cases in which methods are called implicitly by the compiler without appearing in source code. What should this code print?

class Funny {
    public Funny Select(Func<object, object> d,
                        [CallerLineNumber]int line = 0) {
        Console.WriteLine(line + ": " + d(d));
        return this;
    }
}

var r = (from x in new Funny() 
         let y = 1 
         select 2);

This code contains two implicit calls to the Select method that don’t have a clear source line (it gets worse for more complicated LINQ queries)

In fact, it is possible to have an implicit method call with no corresponding source code at all.

Consider this code:

class Loggable {
    public Loggable([CallerLineNumber] int line = 0) { }
}
class SomeClass : Loggable { }

The compiler will implicitly generate a constructor for SomeClass that calls the base Loggable constructor with its default parameter value. What line should it pass? In fact, if SomeClass is a partial class that is defined in multiple files, it isn’t even clear what [CallerFileName] should pass.

Also, what should happen in the unlikely case that a [CallerLineNumber] method is called on line 3 billion (which would overflow an int)? (This would be easier to test on Roslyn with a fake stream source) Should it give an integer overflow compile-time error? If [CallerLineNumber] also supports byte and short parameters, this scenario will be more likely to happen in practice.

Next time: [CallerMemberName]

Subtleties of the new Caller Info Attributes in C# 5

Thu, 06 Oct 2011 13:02:00 -0700

UPDATE: Now that the Visual Studio 11 beta has shipped with this feature implemented, I wrote a separate blog post exploring how it actually behaves in these corner cases.

C# 5 is all about asynchronous programming. However, in additional to the new async features, the C# team managed to slip in a much simpler feature: Caller Info Attributes.

Since C#’s inception, developers have asked for __LINE__ and __FILE__ macros like those in C and C++. Since C# intentionally does not support macros, these requests have not been answered. Until now.

C# 5 adds these features using attributes and optional parameters. As Anders Hejlsberg presented at //Build/, you can write

public static void Log(string message,
    [CallerFilePath] string file = "",
    [CallerLineNumber] int line = 0,
    [CallerMemberName] string member = "")
{
    Console.WriteLine("{0}:{1} – {2}: {3}", 
                      file, line, member, message);

}

If this method is called in C# 5 without specifying the optional parameters, the compiler will insert the file name, line number, and containing member name instead of the default values. If it’s called in older languages that don’t support optional parameters, those languages will pass the default values, like any other optional method.

These features look trivial at first glance. However, like most features, they are actually more complicated to design in a solid and robust fashion. Here are some of the less obvious issues that the C# team needed to deal with when creating this feature:

Disclaimer: I am basing these posts entirely on logical deduction. I do not have access to a specification or implementation of this feature; all I know is what Anders announced in his //Build/ presentation. However, the C# team would have needed to somehow deal with each of thee issues. Since these attributes are not yet supported by any public CTP, I can’t test my assumptions

To start with, they needed to create new compiler errors if the attribute is applied to an incorrectly-typed parameter, or a non-optional parameter. Creating compiler errors is expensive; they need to be documented, tested, and localized into every language supported by C#.

The attributes should perhaps support nullable types, or parameter types with custom implicit conversions to int or string. (especially long or short)

If a method with these attributes is called in an expression tree literal (a lambda expression converted to an Expression<TDelegate>), the compiler would need to insert ConstantExpressions with the actual line number or other info to pass as the parameter.

In addition to being supported in methods, the attributes should also be supported on parameters for delegate types, allowing you to write

delegate void LinePrinter([CallerLineNumber] int line = 0);

LinePrinter printer = Console.WriteLine;
printer();

Each of the individual attributes also has subtle issues.

[CallerFilePath] seems fairly simple. Any function call must happen in source code, in a source file that has a path. However, it needs to take into account #line directives, so that, for example, it will work as expected in Razor views. I don’t know what it does inside #line hidden, in which there isn’t a source file.

Next time: [CallerLineNumber]

Using a default controller in ASP.Net MVC

Mon, 19 Sep 2011 02:01:00 -0700

One common question about ASP.Net MVC is how to make “default” controller.

Most websites will have a Home controller with actions like About, FAQ, Privacy, or similar pages. Ordinarily, these actions can only be accessed through URLs like ~/Home/About. Most people would prefer to put these URLs directly off the root: ~/About, etc.

Unfortunately, there is no obvious way to do that in ASP.Net MVC without making a separate route or controller for each action.

You cannot simply create a route matching "/{action}" and map it to the Home controller, since such a route would match any URL with exactly one term, including URLs meant for other controllers. Since the routing engine is not aware of MVC actions, it doesn’t know that this route should only match actions that actually exist on the controller.

To make it work, we can add a custom route constraint that forces this route to only match URLs that correspond to actual methods on the controller.

To this end, I wrote an extension method that scans a controller for all action methods and adds a route that matches actions in that controller. The code is available at gist.github.com/1225676. It can be used like this:

routes.MapDefaultController<Controllers.HomeController>();

This maps the route "/{action}/{id}" (with id optional) to all actions defined in HomeController. Note that this code ignores custom ActionNameSelectorAttributes. (The built-in [ActionName(…)] is supported)

For additional flexibility, you can also create custom routes that will only match actions in a specific controller. This is useful if you have a single controller with a number of actions that has special route requirements that differ from the rest of your site.

For example:

routes.MapControllerActions<UsersController>(
    name: "User routes",
    url:  "{userName}/{action}"
    defaults: new { action = "Index" }
);

(Note that this example will also match URLs intended for other controllers with the same actions; plan your routes carefully)

XRegExp breaks jQuery Animations

Thu, 15 Sep 2011 01:12:00 -0700

Update: This bug was fixed in XRegExp 1.5.1.
However, as far as I know, there are no released versions of SyntaxHighlighter that contain the fix.

XRegExp is an open source JavaScript library that provides an augmented, extensible, cross-browser implementation of regular expressions, including support for additional syntax, flags, and methods.

It’s used by the popular SyntaxHighlighter script, which is in turn used by many websites (including this blog) to display syntax-highlighted source code on the client. Thus, XRegExp has a rather wide usage base.

However, XRegExp conflicts with jQuery. In IE, any page that includes XRegExp and runs a numeric jQuery animation will result in “TypeError: Object doesn't support this property or method”.

Demo (only fails in IE)

This bug is caused by an XRegExp fix for an IE bug in which the exec method doesn't consistently return undefined for nonparticipating capturing groups. The fix, on line 271, assumes that the parameter passed to exec is a string. This behavior violates the ECMAScript standard (section 15.10.6.2), which states that the parameter to exec should be converted to a string before proceeding.

jQuery relies on this behavior in its animate method, which parses a number using a regex to get the decimal portion. (source)

Thus, calling animate with a number after loading XRegExp will fail in IE when XRegExp tries to call slice on a number.

Fortunately, it is very simple to fix XRegExp to convert its argument to a string first:

if (XRegExp) {
    var xExec = RegExp.prototype.exec;
    RegExp.prototype.exec = function(str) {
        if (!str.slice) 
            str = String(str);
        return xExec.call(this, str);
    };
}

Here is an updated demo that uses this fix and works even in IE.

Clarifying Boolean Parameters, part 2

Wed, 14 Sep 2011 15:30:00 -0700

Part 1 is here

Some languages have better ways to pass boolean parameters. C# 4.0, and all versions of VB, allow parameters to be passed by name. This allows us to write much clearer code:

//C# 4.0:
UpdateLayout(doFullLayout: false)

'VB.Net:
UpdateLayout(doFullLayout:=False)

Without requiring any changes to the function definition, this makes the meaning of the true / false abundantly clear at the call-site.

Javascript offers another interesting alternative. In Javascript, booleans conditions actually check for “truthyness”. The statement if(x) will trigger not just if x is true, but also if x is any “truthy” value, including any object, non-empty string, or non-zero number. Similarly, the expression !x will return false if x is “truthy” and true if x “falsy”.

This means that we can actually use any non-empty string instead of true in Javascript. Note that this will only work if the function checks the value for “truthyness”; it won’t work for code like if (x === true).

Thus, instead of passing true as a boolean, you can pass a string that describes what you’re actually indicating.

For example:

function updatePosition(animate) {
    //Calculate position
    if (animate)
        //...
    else
        //...
}

$(window).resize(function() {
    updatePosition();
});

updatePosition("With animation");

Although this results in much more readable code, it can be difficult to understand for people who aren’t familiar with this trick. If the meaning of the parameter changes, you’ll need to hunt down every place that the function is called and change the string to reflect the new meaning.

Finally, unlike an enum, this does not scale to multiple options. If you need to have more than two options, you should use global variables or objects to simulate an enum, not strings.

Clarifying Boolean Parameters, part 1

Mon, 12 Sep 2011 21:31:00 -0700

Have you ever written code like this:

public void UpdateLayout(bool doFullLayout) {
    //Code
    if (doFullLayout) {
        //Expensive code
    }
    //More code
}

This pattern is commonly used when some operation has a “cheap” mode and an “expensive” mode. Other code will have calls like UpdateLayout(false) and UpdateLayout(true) scattered throughout.

The problem is that this isn’t very obvious for people who aren’t familiar with the codebase. If you take a look at a file you’ve never seen before and see calls like UpdateLayout(false) and UpdateLayout(true) scattered, you’ll have no idea what the true / false means.

The simplest solution is to break it out into two methods: UpdateComplexLayout() and UpdateBasicLayout(). However, if the two different layout modes have intertwined code paths (eg, the code before and after the if above), this either won’t be possible or will lead to ugly duplication of code.

One alternative is to use enums:

public enum LayoutUpdateType {
    Basic,
    Full
}

public void UpdateLayout(LayoutUpdateType type) {
    //Code
    if (type == LayoutUpdateType.Full) {
        //Expensive code
    }
    //More code
}

This way, the callsites are much more descriptive: UpdateLayout(LayoutUpdateType.Full). This also makes it easy to add more update modes in the future should the need arise. However, it makes the callsites much more verbose. When used frequently, this pattern can lead a vast proliferation of enum types that are each only used by one method, polluting the namespace and making more important enums harder to notice.

Next time: Cleverer alternatives

C# is not type-safe

Thu, 08 Sep 2011 02:12:00 -0700

C# is usually touted as a type-safe language. However, it is not actually fully type-safe!

To examine this claim, we must first provide a strict definition of type-safety.
Wikipedia says:

In computer science, type safety is the extent to which a programming language discourages or prevents type errors. A type error is erroneous or undesirable program behavior caused by a discrepancy between differing data types.

To translate this to C#, full type-safety means that any expression that compiles is guaranteed to work at runtime, without causing any invalid cast errors.

Obviously, the cast (and as) operator is an escape hatch from type safety. It tells the compiler that “I expect this value to actually be of this type, even though you can’t prove it. If I’m wrong, I’ll live with that”. Therefore, to be fully type-safe, it must be impossible to get an InvalidCastException at runtime in C# code that does not contain an explicit cast.

Note that parsing or conversion errors (such as any exception from the Convert class) don’t count. Parsing errors aren’t actually invalid cast errors (instead, they come from unexpected strings), and conversion errors from from cast operations inside the Convert class. Also, null reference exceptions aren’t cast errors.

So, why isn’t C# type-safe?

MSDN says that InvalidCastException is thrown in two conditions:

For a conversion from a Single or a Double to a Decimal, the source value is infinity, Not-a-Number (NaN), or too large to be represented as the destination type.

A failure occurs during an explicit reference conversion.

Both of these conditions can only occur from a cast operation, so it looks like C# is in fact type safe.

Or is it?

IEnumerable numbers = new int[] { 1, 2, 3 };

foreach(string x in numbers) 
    ;

This code compiles (!). Running it results in

InvalidCastException: Unable to cast object of type 'System.Int32' to type 'System.String'.

On the foreach line.

Since we don’t have any explicit cast operations (The implicit conversion from int[] to IEnumerable is an implicit conversion, which is guaranteed to succeed) , this proves that C# is not type-safe.

What happened?

The foreach construct comes from C# 1.0, before generics existed. It worked with untyped collections such as ArrayList or IEnumerable. Therefore, the IEnumerator.Current property that gets assigned to the loop variable would usually be of type object. (In fact, the foreach statement is duck-typed to allow the enumerator to provide a typed Current property, particularly to avoid boxing).

Therefore, you would expect that almost all (non-generic) foreach loops would need to have the loop variable declared as object, since that’s the compile-time type of the items in the collection. Since that would be extremely annoying, the compiler allows you to use any type you want, and will implicitly cast the Current values to the type you declared. Thus, mis-declaring the type results in an InvalidCastException.

Note that if the foreach type isn’t compatible at all with the type of the Current property, you will get a compile-time error (just like (string)42 doesn’t compile). Therefore, if you stick with generic collections, you’re won’t get these runtime errors (unless you declare the foreach as a subtype of the item type).

C# also isn’t type-safe because of array covariance.

string[] strings = new string[1];
object[] arr = strings;
arr[0] = 7;

This code compiles, but throws “ArrayTypeMismatchException: Attempted to access an element as a type incompatible with the array.” at run-time.

As Eric Lippert explains, this feature was added in order to be more compatible with Java.

Delegates vs. Function Pointers, Addendum: Multicast Delegates

Tue, 16 Aug 2011 01:02:00 -0700

Until now, I've been focusing on only one of the differences between delegates and function pointers; namely, associated state.
Delegates have one other capability that function pointers do not. A single function pointer can only point to one function. .Net, on the other hand, supports multicast delegates – delegates that point to multiple functions. You can combine two existing delegates using the + operator (or by calling Delegate.Combine) to create a single new delegate instance that points two all of the methods in the original two delegates. This new delegate stores all of the methods from the original two delegates in a private array of delegates called InvocationList (the delegates in this array are ordinary non-multicast delegates that each only point to a single method).

Note that delegates, like strings, are immutable. Adding two delegates together creates a third delegate containing the methods from the first two; the original delegate instances are not affected. For example, writing delegateField += SomeMethod creates a new delegate instance containing the methods originally in delegateField as well as SomeMethod, then stores this new instance in delegateField.

Similarly, the - operator (or Delegate.Remove) will remove the second operand from the first one (again, returning a new delegate instance). If the second operand has multiple methods, all of them will be removed from the final delegate. If some of the methods in the second operand appear multiple times in the original delegate, only the last occurrence of each one will be removed (the one most recently added). The RemoveAll method will remove all occurrences. If all of the methods were removed, it will return null; there is no such thing as an empty delegate instance.

Multicast delegates are not intended to be used with delegates that return values. If you call a non-void delegate that contains multiple methods, it will return the return value of the last method in the delegate. If you want to see the return values of all of the methods, you’ll need to loop over GetInvocationList() and call each delegate individually.

Multicast delegates also don’t play well with the new covariant and contravariant generic delegates in .Net 4.0. You cannot combine two delegates unless their types match exactly, including variant generic parameters.

Function pointers cannot easily be combined the way multicast delegates can. The only way to combine function pointers without cooperation from the code that calls the pointer is to make a function that uses a closure to call all of the function pointers you want to call.

In Javascript, that would look like this:

function combine() {
    var methods = arguments;

    return function() { 
        var retVal;
        for(var i = 0; i < methods.length; i++) 
            retVal = methods[i].apply(this, arguments);
        return retVal;
    };
}

Tracking Event Handler Registrations

Fri, 29 Jul 2011 19:58:00 -0700

When working with large .Net applications, it can be useful to find out where event handlers are being registered, especially in an unfamiliar codebase.

In simple cases, you can do this by right-clicking the event definition and clicking Find All References (Shift+F12). This will show you every line of code that adds or removes a handler from the event by name. For field-like (ordinary) events, this will also show you every line of code that raises the event.

However, this isn’t always good enough. Sometimes, event handlers are not added by name. The .Net data-binding infrastructure, as well as the CompositeUI Event Broker service, will add and remove event handlers using reflection, so they won’t be found by Find All References. Similarly, if an event handler is added by an external DLL, Find All References won’t find it.

For these scenarios, you can use a less-obvious trick. As I described last time, adding or removing an event handler actually executes code inside of an accessor method. Like any other code, we can set a breakpoint to see where the code is executed.

For custom events, this is easy. Just add a breakpoint in the add and/or remove accessors and run your program. Whenever a handler is added or removed, the debugger will break into the accessor, and you can look at the callstack to determine where it’s coming from.

However, most events are field-like, and don’t have actual source code in their accessor methods. To set a breakpoint in a field-like event, you need to use a lesser-known feature: function breakpoints (Unfortunately, this feature is not available in Visual Studio Express). You can click Debug, New Breakpoint, Break at Function (Ctrl+D, N) to tell the debugger to pause whenever a specific managed function is executed.

To add a breakpoint at an event accessor, type Namespace.ClassName.add_EventName. To ensure that you entered it correctly, open the Debug, Breakpoints window (Ctrl+D, B) and check that the new breakpoint says break always (currently 0) in the Hit Count column. If it doesn’t say (currently 0), then either the assembly has not been loaded yet or you made a typo in the location (right-click the breakpoint and click Location).

About .Net Events

Fri, 29 Jul 2011 02:14:00 -0700

A .Net event actually consists of a pair of accessor methods named add_EventName and remove_EventName. These functions each take a handler delegate, and are expected to add or remove that delegate from the list of event handlers.

In C#, writing public event EventHandler EventName; creates a field-like event. The compiler will automatically generate a private backing field (also a delegate), along with thread-safe accessor methods that add and remove handlers from the backing field (like an auto-implemented property). Within the class that declared the event, EventName refers to this private backing field. Thus, writing EventName(...) in the class calls this field and raises the event (if no handlers have been added, the field will be null).

You can also write custom event accessors to gain full control over how handlers are added to your events. For example, this event will store and trigger handlers in reverse order:

void Main()
{
    ReversedEvent += delegate { Console.WriteLine(1); };
    ReversedEvent += delegate { Console.WriteLine(2); };
    ReversedEvent += delegate { Console.WriteLine(3); };

    OnReversedEvent();
}

protected void OnReversedEvent() {
    if (reversedEvent != null)
        reversedEvent(this, EventArgs.Empty);
}

private EventHandler reversedEvent;
public event EventHandler ReversedEvent {
    add {
        reversedEvent = value + reversedEvent;
    }
    remove {
        reversedEvent -= value;
    }
}

This add accessor uses the non-commutative delegate addition operator to prepend each new handler to the delegate field containing the existing handlers. The raiser method simply calls the combined delegate in the private field. (which is null if there aren’t any handlers)

Note that this code is not thread-safe. If two threads add a handler at the same time, both of them will read the original storage field, add their respective handlers to create a new delegate instance, then write this new delegate back to the field. The thread that writes back to the field last will overwrite the changes made by the other thread, since it never saw the other thread’s handler (this is the same reason that x += y is not thread-safe). The accessors generated by the compiler are threadsafe, either by using lock(this) (C# 3 or earlier) or a lock-free threadsafe implementation (C# 4). For more details, see this series of blog posts.

This example is rather useless. However, there are better reasons to create custom event accessors. WinForms controls store their events in a special EventHandlerList class to save memory. WPF controls create events using the Routed Event system, and store handlers in special storage in DependencyObject. Custom event accessors can also be used to perform validation or logging.

Creating Local Extension Methods

Wed, 27 Jul 2011 02:08:00 -0700

Sometimes, it can be useful to make an extension method specifically for a single block of code. Unfortunately, since extension methods cannot appear in nested classes, there is no obvious way to do that.

Instead, you can create a child namespace containing the extension method. In order to limit the extension method’s visibility to a single method, you can put that method in a separate namespace block. This way, you can add a using statement to that namespace alone.

For example:

namespace Company.Project {
    partial class MyClass {
        ...
    }
}
namespace Company.Project {
    using MyClassExtensions;
    namespace MyClassExtensions {
        static class Extensions {
            public static string Name<T>(this T obj) {
                if (default(T) == null && Equals(obj, default(T)))
                    return "(null " + typeof(T) + ")";
                return obj.GetType() + ": " + obj.ToString() 
                     + "{declared as " + typeof(T) + "}";
            }
        }
    }
    partial class MyClass {
        void DoSomething() {
            object x = new DateTime();
            string name = x.Name();
        }
    }
}

Since the using MyClassExtensions statement appears inside the second namespace block, the extension methods are only visible within that block. Code that uses these extension method can appear in this second block, while the rest of the class can go in the original namespace block without the extension methods.

This technique should be avoided where possible, since it leads to confusing and non-obvious code. However, there are situations in which this can make some code much more readable.

Don’t modify other controls during a WPF layout pass

Tue, 26 Jul 2011 02:18:00 -0700

Unlike WinForms or native Win32 development, WPF provides a rich layout model which allows developers to easily create complicated UIs that resize to fit their contents or the parent window.

However, when developing custom controls, it can be necessary to layout child controls manually by overriding the MeasureOverride and ArrangeOverride methods. To quote MSDN,

Measure allows a component to determine how much size it would like to take. This is a separate phase from Arrange because there are many situations where a parent element will ask a child to measure several times to determine its optimal position and size. The fact that parent elements ask child elements to measure demonstrates another key philosophy of WPF – size to content. All controls in WPF support the ability to size to the natural size of their content. This makes localization much easier, and allows for dynamic layout of elements as things resize. The Arrange phase allows a parent to position and determine the final size of each child.

Overriding these methods gives your custom control full power over the layout of its child element(s).

Be careful what you do when overriding these methods. Any code in MeasureOverride or ArrangeOverride runs during the WPF layout passes. in these methods, you should not modify any part of the visual tree outside of the control you’re overriding in. If you do, you’ll be changing the visuals between Measure() and Arrange(), which will have unexpected results.

It is safe to modify your own child controls during the layout pass. Before you call Measure() on a child control, its layout pass has not started. Therefore, any changes will be seen by the child’s layout code. Similarly, after you Arrange() a child control, its layout pass is finished, so it is safe to modify again (although you may end up triggering another layout pass to see the changes).

If you do need to modify an outside control during the layout pass, you should call Dispatcher.BeginInvoke() to run code asynchronously during the next message loop. This way, your code will run after the layout pass finishes, and it will be able to safely modify whatever it wants.

Note that Measure() can be called multiple times during a single layout pass (if a parent needs to iteratively determine the best fit for a child).

Delegates vs. Function Pointers, part 5: Javascript

Mon, 25 Jul 2011 00:33:00 -0700

This is part 5 in a series about state and function pointers; part 1 is here.

Last time, we saw how C# 2 supports closures by compiling anonymous functions into member functions of a special class that holds local state from the outer function.

Unlike the languages we’ve looked at before, Javascript has had closures baked in to the languages since its inception. My standard example can be achieved very simply in Javascript:

var x = 2;
var numbers = [ 1, 2, 3, 4 ];
var hugeNumbers = numbers.filter(function(n) { return n > x; });

This code uses the Array.filter method, new to Javascript 1.6, to create a new array with those elements from the first array that pass a callback. The function expression passed to filter captures the x variable for use inside the callback.

This looks extremely similar to the C# 2.0 version from last time. However. under the covers, it’s rather different.

Like .Net managed instance methods, all Javascript functions take a hidden this parameter. However, unlike .Net, Javascript does not have delegates. There is no (intrinsic) way to bind an object to the this parameter the way a .Net closed delegate does. Instead, the this parameter comes from the callsite, depending on how the function was called. Therefore, we cannot pass state in the this parameter the way we did in C#.

Instead, all Javascript function expressions capture the variable environment of the scope that they are declared in as a hidden property of the function. Therefore, a function can reference local variables from its declaring scope. Unlike C#, which binds functions to their parent scopes using a field in a separate delegate object that points to the function, Javascript functions have their parent scopes baked in to the functions themselves.

Javascript doesn’t have separate delegate objects that can hold a function and a this parameter. Instead, the value of the this parameter is determined at the call-site, depending on how the function was called. This is a common source of confusion to inexperienced Javascript developers.

To simulate closed delegates, we can make a method that takes a function as well as a target object to call it on, and returns a new function which calls the original function with this equal to the target parameter. That sounds overwhelmingly complicated, but it’s actually not that hard:

function createDelegate(func, target) {
    return function() { 
        return func.apply(target, arguments);
    };
}

var myObject = { name: "Target!"};
function myMethod() {
    return this.name;
}

var delegate = createDelegate(myMethod, myObject);
alert(delegate());

This createDelegate method returns a function expression that captures the func and target parameters, and calls func in the context of target. Instead of storing the target in a property of a Delegate object (like .Net does), this code stores it in the inner function expression’s closure.

Javascript 1.8.5 provides the Function.bind method, which is equivalent to this createDelegate method, with additional capabilities as well. In Chrome, Firefox 4, and IE9, you can write

var myObject = { name: "Target!"};
function myMethod() {
    return this.name;
}

var delegate = myMethod.bind(myObject);
alert(delegate());

For more information, see the MDN documentation.

Delegates vs. Function Pointers, part 4: C# 2.0+

Sun, 19 Jun 2011 19:49:00 -0700

This is part 4 in a series about state and function pointers; part 1 is here.

Last time, we saw that it is possible to pass local state with a delegate in C#. However, it involves lots of repetitive single-use classes, leading to ugly code.

To alleviate this tedious task, C# 2 supports anonymous methods, which allow you to embed a function inside another function. This makes my standard example much simpler:

//C# 2.0
int x = 2;
int[] numbers = { 1, 2, 3, 4 };

int[] hugeNumbers = Array.FindAll(
    numbers, 
    delegate(int n) { return n > x; }
);



//C# 3.0
int x = 2;
int[] numbers = { 1, 2, 3, 4 };

IEnumerable<int> hugeNumbers = numbers.Where(n => n > x);

Clearly, this is much simpler than the C# 1.0 version from last time. However, anonymous methods and lambda expressions are compile-time features; the CLR itself is not aware of them. How does this code work? How can an anonymous method use a local variable from its parent scope?

This is an example of a closure – a function bundled together with external variables that the function uses. The C# compiler handles this the same way that I did manually last time in C# 1: it generates a class to hold the function and the variables that it uses, then creates a delegate from the member function in the class. Thus, the local state is passed as the delegate’s this parameter.

To see how the C# compiler implements closures, I’ll use ILSpy to decompile the more-familiar C# 3 version: (I simplified the compiler-generated names for readability)

[CompilerGenerated]
private sealed class ClosureClass {
    public int x;
    public bool Lambda(int n) {
        return n > this.x;
    }
}
private static void Main() {
    ClosureClass closure = new ClosureClass();
    closure.x = 2;
    int[] numbers = { 1, 2, 3, 4 };
    IEnumerable<int> hugeNumbers = numbers.Where(closure.Lambda);
}

The ClosureClass (which was actually named <>c__DisplayClass1) is equivalent to the GreaterThan class from my previous example. It holds the local variables used in the lambda expression. Note that this class replaces the variables – in the original method, instead a local variable named x, the compiler uses the public x field from the ClosureClass. This means that any changes to the variable affect the lambda expression as well.

The lambda expression is compiled into the Lambda method (which was originally named <Main>b__0). It uses the same field to access the local variable, sharing state between the original outer function and its lambda expression.

Next time: Javascript

Open Delegates vs. Closed Delegates

Tue, 14 Jun 2011 14:49:00 -0700

.Net supports two kinds of delegates: Open delegates and closed delegates.

When you create a delegate that points to an instance method, the instance that you created it from is stored in the delegate’s Target property. This property is passed as the first parameter to the method that the delegate points to. For instance methods, this is the implicit this parameter; for static methods, it's the method's first parameter. These are called closed delegates, because they close over the first parameter and bring it into the delegate instance.

It is also possible to create open delegates which do not pass a first parameter. Open delegates do not use the Target property; instead, all of the target method’s parameters are passed from the delegate’s formal parameter list, including the first parameter. Therefore, an open delegate pointing to a given method must have one parameter more than a closed delegate pointing to the same method. Open delegates are usually used to point to static methods. When you make a delegate pointing to a static method, you (generally) don't want the delegate to hold a first parameter for the method.

In addition to these two normal cases, it is also possible (in .Net 2.0 and later) to create open delegates for instance methods and to create closed delegates for static methods. With one exception, C# doesn’t have any syntactical support for these unusual delegates, so they can only be created by calling the CreateDelegate method.

Open delegates by calling the CreateDelegate overload that doesn’t take a target parameter. Before .Net 2.0, this function could only be called with a static method. In .Net 2.0, you can call this function with an instance method to create an open delegate. Such a delegate will call use its first parameter as this instead of its Target field.

As a concrete example, consider the String.ToUpperInvariant() method. Ordinarily, this method takes no parameters, and operates on the string it’s called on. An open delegate pointing to this instance method would take a single string parameter, and call the method on that parameter.

For example:

Func<string> closed = new Func<string>("a".ToUpperInvariant);
Func<string, string> open = (Func<string, string>)
    Delegate.CreateDelegate(
        typeof(Func<string, string>),
        typeof(string).GetMethod("ToUpperInvariant")
    );

closed();     //Returns "A"
open("abc");  //Returns "ABC"

Closed delegates are created by calling the CreateDelegate overload that takes a target parameter. In .Net 2.0, this can be called with a static method and an instance of that method’s first argument type to create a closed delegate that calls the method with the given target as its first parameter. Closed delegates curry the first parameter from the target method. For example:

Func<object, bool> deepThought = (Func<object, bool>)
    Delegate.CreateDelegate(
        typeof(Func<object, bool>),
        2,
        typeof(object).GetMethod("Equals", BindingFlags.Static | BindingFlags.Public)
    );

This code curries the static Object.Equals method to create a delegate that calls Equals with 2 and the delegate’s single parameter). It’s equivalent to x => Object.Equals(2, x). Note that since the method is (generally) not a member of the target object’s type, we need to pass an actual MethodInfo instance; a name alone isn’t good enough.

Note that you cannot create a closed delegate from a static method whose first parameter is a value type, because, unlike all instance methods, static delegates that take value types receive their parameters by value, not as a reference. For more details, see here and here.

C# 3 added limited syntactical support for creating closed delegates. You can create a delegate from an extension method as if it were an instance method on the type it extends. For example:

var allNumbers = Enumerable.Range(1, Int32.MaxValue);
Func<int, IEnumerable<int>> countTo = allNumbers.Take;

This code creates an IEnumerable<int> containing all positive integers, then creates a closed delegate that curries this sequence into the static Enumerable.Take<T>(IEnumerable<T>) method.

Except for extension methods, open instance delegates and closed static delegates are rarely used in actual code. However, it is important to understand how ordinary open and closed delegates work, and where the target object ends up for instance delegates.

Delegates vs. Function Pointers, part 3: C# 1.0

Mon, 13 Jun 2011 21:29:00 -0700

This is part 3 in a series about state and function pointers; part 1 is here.

Last time, we saw that it is impossible to bundle context along with a function pointer in C.

In C#, it is possible to fully achieve my standard example. In order to explain how this works behind the scenes, I will limit this post to C# 1.0 and not use a lambda expression. This also means no LINQ, generics, or extension methods, so I will, once again, need to write the filter method myself.

delegate bool IntFilter(int num);

static ArrayList Filter(IEnumerable source, IntFilter filter) {
    ArrayList retVal = new ArrayList();

    foreach(int i in source)
        if (filter(i))
            retVal.Add(i);
            
    return retVal;
}

class GreaterThan {
    public GreaterThan(int b) {
        this.bound = b;
    }
    int bound;
    public bool Passes(int num) {
        return num > bound;
    }
}

int x = 2;
int[] numbers = { 1, 2, 3, 4 };
Filter(numbers, new IntFilter(new GreaterThan(x).Passes));

Please excuse the ugly code; in order to be true to C# 1.0, I can’t just write an iterator, and I can’t implicitly create the delegate.

This code creates a class called GreaterThan that holds the Passes method and the state used by the method. I create a delegate to pass to Filter out of the Passes method for an instance of the GreaterThan class from the local variable.

To understand how this works, we need to delve further into delegates. .Net delegates are more than just type-safe function pointers. Unlike the function pointers we've looked at, delegates include state – the Target property. This property stores the object to pass as the hidden this parameter to the method (It's actually somewhat more complicated than that; I will describe open and closed delegates at a later date). My example creates a delegate instance in which the Target property points to the GreaterThan instance. When I call the delegate, the Passes method is called on the correct instance from the Target property, so it can read the bound field.

Next time, we'll see how the C# 2.0+ compilers generate all this boilerplate for you using anonymous methods and lambda expressions.

Delegates vs. Function Pointers, part 2: C

Wed, 01 Jun 2011 21:08:00 -0700

This is part 2 in a series about state and function pointers; part 1 is here.

Unlike most other languages, it is not possible to include any form of state in a function pointer in C. Therefore, it is impossible to fully implement closures in C without the cooperation of the call-site and/or the compiler.

To illustrate what this means in practice, I will refer to my standard example, using a filter function to find all elements in an array that are greater than x. Since C doesn’t have any such function, I’ll write one myself, inspired by qsort:

void* filter(void* base, 
             size_t count, 
             size_t elemSize, 
             int (*predicate)(const void *)) {
    //Magic...
}

With this function, one can easily find all elements that are greater than some constant:

int predicate(const void* item) {
        return *(int*)item > 2;
}

int arr[] = { 1, 2, 3, 4 };
filter(arr, 4, sizeof(int), &predicate);

However, if we want to replace the hard-coded 2 with a local variable, it becomes more complicated. Since function pointers, unlike delegates, do not have state, there is no simple way to pass the local variable to the function.

The most obvious answer is to use a global variable:

static int minimum;
int predicate(const void* item) {
        return *(int*)item > minimum;
}

int arr[] = { 1, 2, 3, 4 };
minimum = x;
filter(arr, 4, sizeof(int), &predicate);

However, this code will fail horribly if it’s ever run on multiple threads. In simple cases, you can work around that by storing the value in a global thread-local variable (or in fiber-local storage for code that uses fibers). Because each thread will have its own global, the threads won’t conflict with each-other. Obviously, this won’t work if the callback might be run on a different thread.

However, even if everything is running on one thread, this still isn’t enough if the code can be re-entrant, or if the callback might be called after the original function call finishes. This technique assumes that exactly one callback (from a single call to the function that accepted the callback) will be used at any given time. If the callback might be invoked later (eg, a timer, or a UI handler), this method will break down, since all of the callbacks will share the same global.

In these more complicated cases, one can only pass any state with the cooperation of the original function. In this (overly simple) example, the filter method can be modified to take a context parameter and pass it to the predicate:

void* filter(void* base, 
             size_t count,
             size_t elemSize, 
             int (*predicate)(const void *, const void *), 
             const void* context) {
        //More magic...
}
int predicate(const void* item, const void* context) {
        return *(int*)item > *(int*)context;
}

int arr[] = { 1, 2, 3, 4 };
filter(arr, 4, sizeof(int), &predicate, &x);

To pass around more complicated state, one could pass a pointer to a struct as the context.

Next Time: C# 1.0

Delegates vs. Function Pointers, part 1

Wed, 01 Jun 2011 20:21:00 -0700

Most languages – with the unfortunate exception of Java – allow functions to be passed around as variables. C has function pointers, .Net has delegates, and Javascript and most functional programming languages treat functions as first class objects.

There is a fundamental difference between C-style function pointers vs. delegates or function objects. Pure function pointers cannot hold any state other than the function itself. In contrast, delegates and function objects do store additional state that the function can use.

To illustrate this difference, I will use a simple example. Most programming environments have a filter function that takes a collection and a predicate (a function that takes an item from the collection and returns a boolean), and returns a new collection with those items from the original collection that match the predicate. In C#, this is the LINQ Where method; in Javascript, it’s the Array.filter method (introduced by Javascript 1.6).

Given a list of numbers, you can use such a method to find all numbers in the list that are greater than a local variable x. However, in order to do that, the predicate have access to x. In subsequent posts, I’ll explore how this can be done in different languages.

Html.ForEach in Razor

Wed, 11 May 2011 02:56:00 -0700

Many people write ForEach extension methods for MVC WebForms views, which take a sequence and a delegate to turn each item in the sequence into HTML.

For example:

public static void ForEach<T>(this HtmlHelper html,
                              IEnumerable<T> set, 
                              Action<T> htmlWriter) {
    foreach (var item in set) {
        htmlWriter(item);
    }
}

(The unused html parameter allows it to be called as an extension method like other HTML helpers)

This code can be called like this:

<ul>
    <% Html.ForEach(
            Enumerable.Range(1, 10),
            item => { %> <li><%= item %></li> <% }
    ); %>
</ul>

This code creates a lambda expression that writes markup to the page’s response stream by ending the code block inside the lambda body. Neither the lambda expression nor the ForEach helper itself return anything; they both write directly to the response.

The List<T>.ForEach method can be called exactly the same way.

In Razor views, this method cannot easily be called directly, since Razor pages cannot put markup inside of expressions. You can use a workaround by creating an inline helper and calling it immediately, but it would be better to rewrite the ForEach method to take an inline helper directly.

The naïve way to do that is like this:

public static IHtmlString ForEachSimple<T>(
        this HtmlHelper html,
        IEnumerable<T> set,
        Func<T, HelperResult> htmlCreator
    ) {
    return new HtmlString(String.Concat(set.Select(htmlCreator)));
}

The htmlCreator delegate, which can be passed as an inline helper, returns a HelperResult object containing the markup generated for an item.

This code uses LINQ to call htmlCreator on each item in the set (the Select call), then calls String.Concat to combine them all into one giant string. (String.Concat will call ToString on each HelperResult, which will return the generated markup) We could also call String.Join to put a separator, such as a newline, between every two items.

Finally, it returns an HtmlString to prevent Razor from escaping the returned HTML.

It’s equivalent to the following code using a StringBuilder (this is what String.Concat does internally)

var builder = new StringBuilder();
foreach (var item in set) {
    HelperResult result = htmlCreator(item);
    builder.Append(result.ToString());
}
return new HtmlString(builder.ToString());

This method can be called like this:

<ul>
    @Html.ForEachSimple(
        Enumerable.Range(1, 10),
        @<li>@item</li>
    );
</ul>

The problem with this approach is that it combines all of the content into a giant string. If there are a large number of items, or if each item will have a large amount of markup, this can become (a little bit) slow. It would be better to write each item directly to the caller’s response stream, without assembling any giant strings. This is where HelperResult comes in.

The HelperResult class allows its caller to pass a TextWriter to the WriteTo method, and the helper delegate will write directly to this TextWriter. I can take advantage of this to write a ForEach extension that doesn’t build any strings, by returning a HelperResult instead of a regular IHtmlString.

public static HelperResult ForEachFast<T>(
        this HtmlHelper html,
        IEnumerable<T> set,
        Func<T, HelperResult> htmlCreator
    ) {
    return new HelperResult(tw => {
        foreach (var item in set) {
            htmlCreator(item).WriteTo(tw);
        }
    });
}

This version creates a HelperResult with a delegate that writes each of its items in turn to the TextWriter.

Creating Markup Actions in Razor

Mon, 02 May 2011 18:06:00 -0700

Razor’s inline helpers allow you to create lambda expression that return markup (as a HelperResult). However, there is no simple way to create a lambda expression that writes HTML directly to the page (instead of returning it).

In ASPX pages, one can simply put the beginning and end of a lambda expression in expression blocks, then put markup in the middle.

For example, this code creates a delegate that writes a <b> tag to the page:

<%
    Action pageWriter = () => {%><b>I'm from a lambda!</b><%};
    pageWriter();
    pageWriter();
    pageWriter();
%>

Calling the pageWriter delegate will write directly to the HTTP response stream.

By contrast, Razor inline expressions return their markup. To do this in a Razor page, one would write

@{
    Func<object, HelperResult> htmlMaker
         = @<b>I'm from a lambda!</b>;
    @htmlMaker(null)    //Note @ sign
    @htmlMaker(null)    //Note @ sign
    @htmlMaker(null)    //Note @ sign
}

Calling htmlMaker without an @ sign will return HTML, but won’t write anything to the page.

When working with libraries designed for ASPX pages, it can be necessary to create ASPX-style inline helpers that write to the page instead of returning a value. You can do that by creating an inline helper lambda and passing it to the Write method:

@{
    Action pageWriter = () => Write(new Func<object, HelperResult>(
         @<b>I'm from a lambda!</b>
    )(null));

    pageWriter();
    pageWriter();
    pageWriter();
}

Like the ASPX version, pageWriter now writes directly to the page and does not return anything.

This code can be made simpler by wrapping it in a separate method:

Action MakeAction(Func<object, HelperResult> inlineHelper) {
    return () => Write(inlineHelper(null));
}

This method takes an inline helper and returns an Action that writes the helper’s output to the page. Since it needs to call the page’s Write method (to use the page’s output stream), this method must be defined in the WebPageBase instance, either in an @functions block or in a common base class.

Any code in the inline helper will execute each time the resulting Action is called.

It can be called like this:

@{
    Action pageWriter2 = MakeAction(@<b>I'm from a lambda!</b>);

    pageWriter();
    pageWriter();
    pageWriter();
}

This code is equivalent to the previous sample, but it’s much simpler.

One can also write write a version that takes a parameter:

Action<T> MakeAction<T>(Func<T, HelperResult> inlineHelper) {
    return param => Write(inlineHelper(param));
}

Note that the type parameter must be specified explicitly; C# does not infer type parameters from the method’s return type.

Dissecting Razor, part 9: Inline Helpers

Fri, 29 Apr 2011 14:08:00 -0700

In addition to normal and static helpers, Razor supports inline helpers, (also known as templates), which allow you to create lambda expressions that return markup.

An inline helper is created by writing @<tag>Content</tag> as an expression. This creates a lambda expression that takes a parameter called item and returns a HelperResult object containing the markup.

Inline helpers are used to create functions that take markup as parameters. For example, you might make an IfLoggedOn helper that displays content if there is a logged-in user, but shows a login link to anonymous users. To pass the content to the helper, you can use an inline helper:

@helper IfLoggedOn(Func<MembershipUser, HelperResult> content) {
    var currentUser = Membership.GetUser();
    if (currentUser == null) {
        <i>
            To use this content, you must 
            <a href="@Href("~/Login")">log in</a>.
        </i>
    } else {
        @content(currentUser)
    }
}

You can call this method with an inline helper like this:

@IfLoggedOn(
    @<form action="@Href("Add-Comment")" method="post">
        @Html.TextArea("Comment", "Type a comment")
        <input type="submit" name="Submit" value="Add Comment" />
    </form>
)

The @content(currentUser) call in helper method is translated by the Razor compiler into a call to the overload of the Write method that takes a HelperResult (returned from the delegate). This overload writes the content of the HelperResult to the page without escaping it (Just like a call to a normal helper method). Functions that take inline helpers can also get the text rendered by the helper by calling ToString() on the HelperResult.

The call to the helper method is compiled to the following C# code:

Write(IfLoggedOn(
item => new System.Web.WebPages.HelperResult(__razor_template_writer => {

    WriteLiteralTo(@__razor_template_writer, "    ");
    WriteLiteralTo(@__razor_template_writer, "<form action=\"");

    WriteTo(@__razor_template_writer, Href("Add-Comment"));

    WriteLiteralTo(@__razor_template_writer, "\" method=\"post\">\r\n        ");

    WriteTo(@__razor_template_writer, Html.TextArea("Comment", "Type a comment"));

    WriteLiteralTo(@__razor_template_writer, "\r\n   ...  </form>\r\n");

})));

The IsLoggedOn method is passed a lambda expression that takes an item parameter and returns a new HelperResult. The HelperResult is constructed exactly like a normal helper, except that it’s in a lambda expression instead of a normal method.

Like normal helpers, then markup inside inline helpers can contain arbitrary code blocks and code nuggets. However, inline helpers cannot be nested (since that would create conflicting parameter names)

The item parameter is implicitly typed based on the delegate type that the helper is being passed as (just like normal lambda expressions). This allows inline helpers to accept a parameter:

@IfLoggedOn(@<text>Welcome, @item.UserName!</text>)

(The special <text> tag is used to pass markup without creating an actual HTML tag)

Next Time: Sections

Dissecting Razor, part 8: Static Helpers

Tue, 08 Mar 2011 16:41:00 -0800

Razor helpers can be extremely useful, but they are designed to be used only by the page that created them.

To create reusable helpers, you can create CSHTML pages in the App_Code directory. The WebRazorHostFactory will check for pages in App_Code and create a WebCodeRazorHost instead of the normal WebPageRazorHost.

This happens before the virtual CreateHost method; in order to change this behavior, one must create an inherited RazorBuildProvider and override the CreateHost method; for more details, see the source for more details.

The WebCodeRazorHost compiles helper methods as public static methods by setting the RazorEngineHost.StaticHelpers property to true. It also overrides PostProcessGeneratedCode to remove the Execute() method and make the Application property static.

Static helper pages inherit the HelperPage class, a very stripped-down base class which contains the static WriteTo methods used by the helpers and some static members that expose the Model, Html, and other members from the currently executing page (using the WebPageContext.Current.Page property).

Because the generated Execute() method is removed, all content outside helpers and @functions blocks is not seen by the compiler. It is seen by the Razor parser, so it must contain valid Razor syntax (eg, no stray @ characters).

Here is an example:

@helper Menu(params string[][] items) {
    <ul>
        @foreach (var pair in items) {
            <li><a href="@Href(pair[1])">@pair[0]</a></li>
        }
    </ul>
}

This text and @code is not compiled.

This helper can then be called from any other code by writing PageName.Menu(...), where PageName is the filename of the CHSTML page in App_Code. In Razor pages, it can be called like any other helper. In normal C# code, it returns a HelperResult instance; call ToString() or WriteTo(TextWriter) to get the HTML source.

For example: (assuming that the helper is defined in ~/App_Code/Helpers.cshtml)

@Helpers.Menu(
    new[] { "Home",    "~/"        },
    new[] { "About",   "~/About"   },
    new[] { "Contact", "~/Contact" }
)

In order to see the source, I need to insert a #error directive inside the helper block; putting it in the page itself will have no effect, since the page is not compiled. Since the contents of the helper block are code, not markup, I don’t need to wrap it in @{ ... }.

The above helper is transformed into the following C#: (As usual, @line directives have been removed)

public class Helpers : System.Web.WebPages.HelperPage {
    public static System.Web.WebPages.HelperResult Menu(params string[][] items) {
        return new System.Web.WebPages.HelperResult(__razor_helper_writer => {

            WriteLiteralTo(@__razor_helper_writer, "    <ul>\r\n");
            foreach (var pair in items) {
                WriteLiteralTo(@__razor_helper_writer, "            <li><a href=\"");
                WriteTo(@__razor_helper_writer, Href(pair[1]));
                WriteLiteralTo(@__razor_helper_writer, "\">");
                WriteTo(@__razor_helper_writer, pair[0]);
                WriteLiteralTo(@__razor_helper_writer, "</a></li>\r\n");
            }
            WriteLiteralTo(@__razor_helper_writer, "    </ul>\r\n");
#error
        });
    }
    public Helpers() {
    }
    protected static System.Web.HttpApplication ApplicationInstance {
        get {
            return ((System.Web.HttpApplication)(Context.ApplicationInstance));
        }
    }
}

Note that the page-level content does not show up anywhere. The helper itself is compiled exactly like a normal helper, except that it’s static.

Although there aren’t any in this example, @functions blocks will also be emitted normally into the generated source.

Next Time: Inline Helpers

Boot Camp just got better

Sun, 06 Mar 2011 16:33:00 -0800

The Boot Camp drivers in the latest generation of MacBook Pros now expose more of the laptop’s hardware to Windows.

This means that Windows can now adjust the screen brightness in the Windows Mobility Center, and when you plug in or unplug the laptop.

The new drivers also expose the laptop’s ambient light sensor via Windows 7’s sensor platform, allowing the screen brightness to be adjusted automatically, and allowing 3rd-party programs to read the brightness.

Dissecting Razor, part 7: Helpers

Mon, 28 Feb 2011 16:08:00 -0800

We’ll continue our trek into Razor’s class-level features with helpers.

Helpers are one of Razor’s unique features. They encapsulate blocks of HTML and server-side logic into reusable page-level methods.

You can define a helper by writing @helper MethodName(parameters) { ... }. Inside the code block, you can put any markup or server-side code. The contents of a helper are parsed as a code block (like the contents of a loop or if block), so any non-HTML-like markup must be surrounded by <text> tags or prefixed by the @: escape.

Here is a simple example:

<!DOCTYPE html>
<html>
    <body>
        @helper NumberRow(int num) {
            var square = num * num;
            <tr>
                <td>@num</td>
                <td>@square</td>
                <td>@(num % 2 == 0 ? "Even" : "Odd")</td>
            </tr>
        }
        <table>
            <thead>
                <tr>
                    <th>Number</th>
                    <th>Square</th>
                    <th>Eveness</th>
                </tr>
            </thead>
            <tbody>
                @for (int i = 0; i < 10; i++) {
                    @NumberRow(i)
                }
            </tbody>
        </table>
    </body>
</html>

Note that code statements (such as the square declaration) can go directly inside the helper body without being wrapped in code blocks – the direct contents of the helper is a code block, not markup. Like any other code block, HTML-like markup is automatically treated as markup instead of code.

Like @functions blocks, helper methods can go anywhere in the source file; the physical location of the block is ignored.

Here is the generated source for the above example: (with blank lines and #line directives stripped for clarity)

public class _Page_Razor_Helpers_cshtml : System.Web.WebPages.WebPage {

    public System.Web.WebPages.HelperResult NumberRow(int num) {
        return new System.Web.WebPages.HelperResult(__razor_helper_writer => {

            var square = num * num;

            WriteLiteralTo(@__razor_helper_writer, "            <tr>\r\n                <td>");
            WriteTo(@__razor_helper_writer, num);

            WriteLiteralTo(@__razor_helper_writer, "</td>\r\n                <td>");
            WriteTo(@__razor_helper_writer, square);

            WriteLiteralTo(@__razor_helper_writer, "</td>\r\n                <td>");
            WriteTo(@__razor_helper_writer, num % 2 == 0 ? "Even" : "Odd");

            WriteLiteralTo(@__razor_helper_writer, "</td>\r\n            </tr>\r\n");

        });
    }
    public _Page_Razor_Helpers_cshtml() {
    }
    protected ASP.global_asax ApplicationInstance {
        get {
            return ((ASP.global_asax)(Context.ApplicationInstance));
        }
    }
    public override void Execute() {
        WriteLiteral("<!DOCTYPE html>\r\n<html>\r\n    <body>\r\n        ");
        WriteLiteral("\r\n        <table>\r\n            <thead>\r\n                <tr>\r\n                   " +
        " <th>Number</th>\r\n                    <th>Square</th>\r\n                    <th>E" +
        "veness</th>\r\n                </tr>\r\n            </thead>\r\n            <tbody>\r\n");

        for (int i = 0; i < 10; i++) {
            Write(NumberRow(i));
        }

        WriteLiteral("            </tbody>\r\n        </table>\r\n    </body>\r\n</html>\r\n");
#error

    }
}

Helpers are compiled as class-level methods that take a parameter set and return a System.Web.WebPages.HelperResult. (This class name is configured by the RazorHostFactory)

Notice the the contents of the helper method are inside a lambda expression that takes a parameter named __razor_helper_writer. This construction allows the helper to write directly to the HTTP response stream instead of assembling a giant string and then writing the string all at once.

The HelperResult constructor takes an Action<TextWriter> which contains the contents of the helper block. The class implements IHtmlString and calls the action from the constructor to generate HTML. However, under normal circumstances, this IHtmlString implementation is never called.

Calls to helper methods (@NumberRow(i)) are passed to the Write(HelperResult) overload. This overload calls HelperResult.WriteTo(writer), which passes the page’s TextWriter directly to the helper’s lambda expression. Thus, the lambda expression can write directly to the page’s output stream, without passing the output as a parameter to the helper method.

Looking inside the helper, we see that all content is passed to WriteTo and WriteLiteralTo methods, as opposed to the Write and WriteLiteral methods used by the rest of the page.

Helper methods cannot call the normal Write* methods since they aren’t necessarily writing to the current output (even though they usually do). Therefore, they call these Write*To methods, which accept the TextWriter as a parameter. These static methods are inherited from the WebPageExecutingBase class; their names are also configured by the RazorHostFactory. The @ in the parameter is a little-used C# syntactictal feature that allows keywords to be used as identifiers; it has nothing to do with Razor’s use of the @ character.

Since helpers are compiled as normal methods, they can do almost anything that a normal method can. However, because their contents are compiled inside a lambda expression, they have some limitations. For example, helpers cannot use ref or out parameters, since they cannot be used inside lambda expressions. Helpers can take params arrays or optional parameters.

Also, Razor’s C# code parser doesn’t support generic helper methods, although there is no reason that they couldn’t be supported in a later release.

The VB.Net code parser also doesn’t explicitly support generic helpers. However, because VB.Net generics use parentheses, a VB.Net helper can declare a generic type parameter instead of a normal parameter list.

For example:

<!DOCTYPE html>
<html>

    <body>
        @Helper PrintType(Of T)
            @GetType(T)
        End Helper
        @PrintType(Of Dictionary(Of Integer, List(Of String)))()
    </body>

</html>

This trick is not very useful.

Next Time: Static Helpers

Dissecting Razor, part 6: Function Blocks

Fri, 18 Feb 2011 02:46:00 -0800

After looking at how Razor’s Execute() method is generated, we will turn to class-level features.

C# Razor pages can define class members inside of @functions { ... } blocks. These are Razor’s equivalent of <script runat="server"> blocks in ASPX pages. VBHTML pages use @Functions ... End Functions instead.

Functions blocks are emitted directly into top of the generated class, regardless of their location in the original source. Unlike code blocks, function blocks cannot contain markup.

Here is a simple example:

<!DOCTYPE html>
<html>
    <body>
        @functions{
            public int GetPageLength() {
                //Don't try this in production.
                return ((StringWriter)this.Output).ToString().Length;
            }
        }
        @GetPageLength() characters have been written so far.
    </body>
</html>
@{ #error }

Note that functions blocks can be defined anywhere, even in the middle of the markup. The location of the block is totally irrelevant.

Here is the generated C# source, with a comment indicating where the block used to be.

public class _Page_Razor_Functions_cshtml : System.Web.WebPages.WebPage {

#line hidden
#line 4 "...\Functions.cshtml"

    public int GetPageLength() {
        //Don't try this in production.
        return ((StringWriter)this.Output).ToString().Length;
    }

#line default
#line hidden

    public _Page_Razor_Functions_cshtml() {
    }

    protected ASP.global_asax ApplicationInstance {
        get {
            return ((ASP.global_asax)(Context.ApplicationInstance));
        }
    }

    public override void Execute() {
        WriteLiteral("<!DOCTYPE html>\r\n<html>\r\n    <body>\r\n        ");
//Here was the functions block
        WriteLiteral("\r\n        ");

#line 10 "...\Functions.cshtml"
        Write(GetPageLength());
#line default
#line hidden
        WriteLiteral(" characters have been written so far.\r\n    </body>\r\n</html>\r\n");

#line 13 "...\Functions.cshtml"
#error

#line default
#line hidden
        WriteLiteral("     ");
    }
}

The contents of the functions block is inserted at the very top of the class, with the familiar #line directives to pretend that it comes from the CSHTML.

Notice that none of the whitespace around the functions block is stripped; you can see the newline and indentation in the @functions line in the first WriteLiteral string, and the newline and indentation after the closing } in the second WriteLiteral string.

Sure enough, the rendered HTML contains an extra blank like:

<!DOCTYPE html>
<html>
    <body>
        
        55 characters have been written so far.
    </body>
</html>

This means that putting a functions block before the <!DOCTYPE> will cause the HTML to start with an ugly blank line. Therefore, it’s best to put functions blocks at the end of the source, where the blank lines won’t matter.

Next Time: Helpers

Dissecting Razor, part 5: Use the Source, Luke

Wed, 16 Feb 2011 16:54:00 -0800

Last time, we saw how basic Razor constructs are translated into C#.

We can see the generated class by adding @{ #error } to the page source. This creates a compiler error in the Execute method, and the resulting Yellow Screen of Death contains a Show Complete Compilation Source: link which will show the generated C# class.

Let’s start with a very simple page:

<!DOCTYPE html>
<html>
    <body>
        1 + 2 = @(1 + 2)<br />
        @{ var source = "<b>bold &amp; fancy</b>"; }
        <code>@source</code> is rendered as
        @(new HtmlString(source))
    </body>
</html>
@{ #error }

This page is rendered like this: (after removing @{ #error })

<!DOCTYPE html>
<html>
    <body>
        1 + 2 = 3<br />
        <code>&lt;b&gt;bold &amp;amp; fancy&lt;/b&gt;</code> is rendered as
        <b>bold &amp; fancy</b>
    </body>
</html>

As expected, the expression @source is automatically escaped. Also notice that the newline and indentation around the code block (@{ var ... }) was not rendered – the Razor parser strips all whitespace surrounding code blocks. This is a welcome improvement over the ASPX view engine.

Now let’s look at how this HTML is generated. This page is transformed into the following C# source:

namespace ASP {
    using System;
    using System.Collections.Generic;
    using System.IO;
    using System.Linq;
    using System.Net;
    using System.Web;
    using System.Web.Helpers;
    using System.Web.Security;
    using System.Web.UI;
    using System.Web.WebPages;
    using System.Web.WebPages.Html;
    using WebMatrix.Data;
    using WebMatrix.WebData;
    public class _Page_Razor_SimplePage_cshtml : System.Web.WebPages.WebPage {

#line hidden
        public _Page_Razor_WriteTo_cshtml() {
        }

        protected ASP.global_asax ApplicationInstance {
            get {
                return ((ASP.global_asax)(Context.ApplicationInstance));
            }
        }

        public override void Execute() {
            WriteLiteral("<!DOCTYPE html>\r\n<html>\r\n    <body>\r\n        1 + 2 = ");

#line 4 "...\SimplePage.cshtml"
            Write(1 + 2);
#line default
#line hidden
            WriteLiteral("<br />\r\n");

#line 5 "...\SimplePage.cshtml"
            var source = "<b>bold &amp; fancy</b>";
#line default
#line hidden
            WriteLiteral("        <code>");

#line 6 "...\SimplePage.cshtml"
            Write(source);
#line default
#line hidden
            WriteLiteral("</code> is rendered as\r\n        ");

#line 7 "...\SimplePage.cshtml"
            Write(new HtmlString(source));
#line default
#line hidden
            WriteLiteral("\r\n    </body>\r\n</html>\r\n");

#line 10 "...\SimplePage.cshtml"
#error
#line default
#line hidden

        }
    }
}

The WebPageRazorEngineHost injects the ApplicationInstance property into the CodeDOM tree; this property allows code in the page to access any custom properties in Global.asax.

As mentioned earlier, the page source is compiled into the Execute() method.

It uses #line directives to pretend that its code is actually in the CSHTML page. This means that code or line numbers appearing in error pages come from the original CSHTML source, making the code easier to find when debugging. The #line hidden directives indicate generated source that did not come from actual code in the CSHTML.

As mentioned last time, literal HTML source is passed to the WriteLiteral method, which is inherited from the WebPageBase class. This method writes its argument to the current output stream (which can vary when making sections). These calls are wrapped in #line hidden because they come from literal text, not code.

The two code blocks (the variable declaration and the #error directive) are copied straight into Execute(), wrapped in #line directives that map them to the actual code lines in the CSHTML.

The code nuggets are passed to the Write method, and are similarly wrapped in #line directives.

Here is a more sophisticated Razor page:

<!DOCTYPE html>
<html>
    <body>
        @{ const int count = 10; }
        <table>
            @for (int i = 0; i < count; i++) {
                <tr>
                    <td>@i</td>
                    <td>@(i * i)</td>
                </tr>
            }
        </table>
    </body>
</html>
@{ #error }

The @for loop is a code block in the form of a control flow statement. Razor’s C# parser is aware of C# control flow structures and parses them as code blocks. (The VB parser does the same thing)

Here is the generated Execute() method, with #line directives removed for clarity:

public override void Execute() {
    WriteLiteral("<!DOCTYPE html>\r\n<html>\r\n    <body>\r\n");

    const int count = 10;
    WriteLiteral("        <table>\r\n");

    for (int i = 0; i < count; i++) {
        WriteLiteral("                <tr>\r\n                    <td>");

        Write(i);
        WriteLiteral("</td>\r\n                    <td>");

        Write(i * i);
        WriteLiteral("</td>\r\n                </tr>\r\n");

    }
    WriteLiteral("        </table>\r\n    </body>\r\n</html>\r\n");
#error
}

Here too, we see that all contiguous chunks of literal HTML are passed to WriteLiteral.

This example has two code blocks – the const declaration and the loop. The for loop code block has HTML inside of it – any HTML markup inside a code block is parsed as normal HTML and passed to WriteLiteral.

Next Time: Function blocks

Dissecting Razor, part 4: Anatomy of a Razor Page

Wed, 09 Feb 2011 14:08:00 -0800

After looking at the various assemblies in the WebPages framework, we will drill into the inner workings of Razor pages.

Razor Side

An ordinary CSHTML page is transformed into a class which inherits the WebPage class. The generator overrides the abstract Execute() method from the to render the page to the HTTP response stream.

Except for class-level directives and constructs (which will be discussed later), all ordinary content in a Razor page end up in the Execute method.

There are three types of normal content: Literals, Code Blocks, and Code Nuggets.

Literals include any normal text. Razor compiles literal text into calls to the WriteLiteral method, with the text as a (correctly-escaped) string parameter. Razor expects this method to write its parameter to the page.

Code Blocks include @{ ... } blocks, as well as control structures. They’re Razor’s equivalent of <% ... %> blocks. The contents of a code block are emitted as-is in the Execute method. Code blocks must contain complete statements, or they’ll result in C# syntax errors.

VBHTML pages use @Code ... End Code blocks instead.

Code Nuggets are @-blocks. They’re Razor’s equivalent of <%: ... %> blocks in an ASPX page. Scott Guthrie describes how these blocks are tokenized. The contents of a code nugget are passed to the Write method, which is expected to HTML-escape its parameter and print it.

WebPages Side

The WebPages framework’s Write method (which comes from the WebPageBase class) takes a parameter of type Object, allowing one to put any expression in a code nugget. It passes its parameter to HttpUtility.HtmlEncode, which will call ToString() and HTML-escape the output. If the parameter is an IHtmlString, HtmlEncode will return its ToHtmlString() method without escaping.

The base class, method names, and default namespaces can be configured in the RazorEngineHost. In addition, custom RazorEngineHosts can override the PostProcessGeneratedCode method to make arbitrary modifications to the generated code.

The WebRazorHostFactory in System.Web.WebPages.Razor.dll can also read default namespaces, a default base type, and a custom host from the <system.web.webPages.razor> section in Web.config:

<system.web.webPages.razor>
    <host factoryType="MyProject.MyWebPageRazorHost" />
    <pages pageBaseType="MyProject.CustomWebPage">
        <namespaces>
            <add namespace="MyProject.SomeNamespace" />
        </namespaces>
    </pages>
</system.web.webPages.razor>

Next Time: Looking at the generated page

Dissecting Razor, part 3: Razor and MVC

Thu, 03 Feb 2011 18:49:00 -0800

Last time, we saw how standalone Razor pages are served.

MVC3 maintains the strict separation between the WebPages framework and the Razor engine.1

Razor Side

Like the WebPages framework, MVC3 interacts with Razor indirectly, by relying on RazorBuildProvider from System.Web.WebPages.Razor.dll. However, MVC3 requires that Razor views inherit its own base class, System.Web.Mvc.WebViewPage.

MVC3 adds a new @model directive, which can be used instead of @inherits to specify a strongly-typed model. This syntax is implemented by customized RazorCodeParsers and RazorCodeLanguages in the System.Web.MVC.Razor namespaces. These classes are invoked by MvcRazorEngineHosts from a custom RazorHostFactory registered in Views/Web.Config:

<system.web.webPages.razor>
    <host factoryType="System.Web.Mvc.MvcWebRazorHostFactory, System.Web.Mvc, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" />
    <pages pageBaseType="System.Web.Mvc.WebViewPage">
        <namespaces>
            <add namespace="System.Web.Mvc" />
            <add namespace="System.Web.Mvc.Ajax" />
            <add namespace="System.Web.Mvc.Html" />
            <add namespace="System.Web.Routing" />
        </namespaces>
    </pages>
</system.web.webPages.razor>

MVC Side

On the MVC side, MVC3 includes a new RazorViewEngine which creates RazorView instances. RazorView inherits the existing BuildManagerCompiledView class, which passes the view’s virtual path to the build manager. RazorView will take the WebViewPage from the build manager, find any matching start pages, and execute the view.

As with the WebPages framework, one can substitute other templating engines. One can register a build provider which compiles classes that inherit WebViewPage, then add a RazorViewEngine to ViewEngines.Engines with additional extensions in its FileExtensions property.

Next time: Inside Razor Pages

Dissecting Razor, part 2: Gluing the pieces together

Wed, 02 Feb 2011 16:46:00 -0800

Last time, we saw that ASP.Net Web Pages are implemented in two independent assemblies. These assemblies are not directly connected to each-other.

Razor Side

System.Web.WebPages.Razor.dll contains the RazorBuildProvider class, which allows ASP.Net’s build system to compile Razor pages. This class uses a WebRazorHostFactory to create WebPageRazorHosts used to process CSHTML (or VBHTML) files into CodeDOM trees. It compiles the CodeDOM tree and returns the generated type(which will typically inherit System.Web.WebPages.WebPage) to the build system.

WebPageRazorHost is coupled to the WebPages framework; it handles the non-standard base types for special pages (StartPage and ApplicationStartPage).

RazorBuildProvider can be configured to use a different WebRazorHostFactory that creates custom WebPageRazorHosts. (more on this later)

WebPages Side

The WebPages framework contains an internal WebPageHttpModule class which runs when an ASP.Net AppDomain is started. It runs any _AppStart files, and hooks the request lifecycle to handle requests for CSHTML (or VBHTML) pages.

Requests for Razor pages are handled by System.Web.WebPages.WebPageHttpHandler. This class passes the page’s virtual path to the build manager and gets a WebPage instance (It assumes that RazorBuildProvider will build the page and give a WebPage instance).

The handler calls the page’s ExecutePageHierarchy method to serve the page to the client. This method runs any _PageStart pages in the page’s parent directories, then executes the page.

The WebPageHttpHandler will also add a custom HTTP header, X-AspNetWebPages-Version: 1.0. This can be disabled by setting WebPageHttpHandler.DisableWebPagesResponseHeader to false.

As mentioned, System.Web.WebPages.dll is not directly tied to the Razor language and engine. One can create a custom build provider which compiles classes that inherit WebPage, then call WebPageHttpHandler.RegisterExtension to tell the WebPages framework to handle requests to the extension, without using the Razor parser.

The source code for this project is part of the same bundle.

Next time: MVC Razor views

Dissecting Razor, part 1: Parts of the framework

Tue, 01 Feb 2011 03:51:00 -0800

Razor involves two distinct components: The Razor engine and the WebPages framework.

The Razor engine, in System.Web.Razor.dll, parses CSHTML (and VBHTML) files into CodeDOM trees. Except for the word Web in project name, the engine has nothing to do with ASP.Net; it doesn’t even reference System.Web.dll. In fact, it targets the .Net Client Profile, and only references mscorlib and System.dll.

The Razor engine is aware of all of Razor’s syntax-level features (code nuggets, sections, helpers), but is not aware of what they mean; it blindly transforms them into function calls.

The Razor engine can be used without ASP.Net for any kind of templating system. This is done by inheriting the RazorEngineHost class to provide default code-generation settings (such as base class and method name), then passing the host to a RazorTemplateParser.

The standard syntax will be annoying when generating non-XML-like content. To avoid this, one can write a custom MarkupParser to define a Razor syntax for a different markup language (such as CSS).

The WebPages framework, in System.Web.Webpages.dll, is a set of classes to use with the Razor parser. It contains the WebPage class which standard Razor pages inherit. This class defines the methods which are blindly called by the Razor parser, such as DefineSection and WriteLiteral. This framework also handles _PageStart and _AppStart pages and contains the HtmlHelper infrastructure.

The WebPages framework is not directly connected to the Razor parser. It could theoretically be used with a different template engine, as long as the template engine emits the correct method calls and class definitions. (more on this later)

The WebPages framework also contains two sets of utility methods. System.Web.Helpers.dll contains miscellaneous utility classes, such as Crypto and WebMail wrappers, plus grid and chart implementations. Microsoft.Web.Helpers.dll contains HTML helper classes which integrate with various third-party services, including Twitter, ReCaptcha, Google Analytics, and more. Most of these helpers can also be used in ordinary ASPX pages.

The source code for all of these projects is available here.

Next time: Gluing it all together.

Modifying HTML strings using jQuery

Mon, 31 Jan 2011 13:55:00 -0800

jQuery makes it very easy to modify a DOM tree. For example, to strip all hyperlinks (<a> tags) from an element, we can write (demo)

$(...).find('a[href]')
      .replaceWith(function() { return this.childNodes });

After getting used to this, one might want to use jQuery to modify HTML contained in a string. Here, however, the naïve approach does not work:

var htmlSource = ...;
$(htmlSource).find('a[href]')
      .replaceWith(function() { return this.childNodes });

This code tries to remove all <a> tags from the HTML contained in the htmlSource string. However, what it actually does is create a detached DOM tree containing the new elements, strip all <a> tags in those elements, and throw the whole thing away. It doesn’t modify the original string. In fact, since the $ function only takes a reference to an immutable string, this approach cannot modify the original string.

Instead, you need to retrieve the source from the DOM tree after modifying it, then assign that source back to the variable.

There is an additional subtlety with this approach. jQuery cannot return the complete HTML source for a collection of elements. Therefore, it is also necessary to wrap the HTML in a dummy element (typically a <div>). One can then call .html() to get the innerHTML of the dummy element, which will contain exactly the desired content

This also eliminates the distinction between root-level elements and nested elements. If the original HTML string contains root-level <a> elements (which aren’t nested in other tags), writing $(htmlSource).find('a') won’t find them, since .find() only searches the descendants of the elements in the jQuery object. By wrapping the HTML in a dummy element, all of the elements in the original content become descendants, and can be returned by .find().

Here, therefore, is the correct way to modify an HTML string using jQuery:

var htmlSource = ...;
var tree = $("<div>" + htmlSource + "</div>");

tree.find('a[href]')
    .replaceWith(function() { return this.childNodes });

htmlSource = tree.html();

Generic Base Classes in ASP.Net MVC

Sun, 30 Jan 2011 23:46:00 -0800

Last time, we saw that there are severe limitations in creating ASPX pages which inherit generic base classes. Many readers were probably wondering how ASP.Net MVC works around this limitation. In ASP.Net MVC views, people write pages like this all the time:

<%@ Page Language="C#" 
    Inherits="ViewPage<IEnumerable<DataLayer.Product>>" %>

ASP.Net MVC includes its own workaround for these limitations. The Web.config file in the Views folder of an ASP.Net MVC project registers a PageParserFilter:

<pages
    validateRequest="false"
    pageParserFilterType="System.Web.Mvc.ViewTypeParserFilter, System.Web.Mvc, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35"
    ...>
    ...
</pages>

PageParserFilter is one of ASP.Net’s lesser-known extensibility points. It can intercept different parts in the parsing process for an ASPX page and modify the page. The MVC framework’s ViewTypeParserFilter will check whether the page’s inherits="" attribute contains ( or a < characters; these characters can only appear in C# or VB.Net generic types, but not in the CLR’s native syntax.

If the inherits="" attribute contains these characters, it will save the attribute’s original value, then replace it with ViewPage (Or ViewMasterPage or ViewUserControl, as appropriate). This way, the rest of the built-in ASP.Net parser will see a normal type name that it knows how to parse. After the page finishes parsing, an internal ControlBuilder registered on MVC’s base types (ViewPage, ViewMasterPage or ViewUserControl) will replace the base type in the generated CodeDOM tree with the original value of the inherits="" attribute.

The one problem with the hack is that it leaves the ASPX parsing engine unaware of the page’s actual base type. Therefore, if you make a page that inherits a generic base class with additional properties, you won’t be able to set those properties in the <%@ Page declaration (since the ASPX parser won’t know about them). If you inherit a non-generic type, this mechanism will not kick in and page properties will work fine.

Generic Base Classes in ASP.Net

Thu, 27 Jan 2011 16:55:00 -0800

ASP.Net pages can inherit from custom classes (as long as they inherit System.Web.UI.Page). This can be useful to add utility functions or shared (code-behind) behaviors to your pages. (Note that you could also use Extension methods or HTTP modules)

However, if you try to inherit a generic base class, it won’t work:

public class DataPage<T> : Page {
    public T Data { get; set; }
}

<%@ Page Language="C#" Inherits="DataPage<string>" %>

This code results in a yellow screen of death, with the parser error, Could not load type 'DataPage<string>'.

This happens because the ASP.Net page parser is unaware of C# generics syntax. The familiar generics syntax (eg, List<int>) is actually a C# innovation and is not used at all in the actual framework. The “native” generics syntax, which is used by Reflection, is markedly different: List`1[Int32] (namespaces omitted for brevity). This name is returned by the Type.AssemblyQualifiedName property.

Since ASP.Net uses reflection APIs to load types, we need to specify generic types using CLR syntax (and with full namespaces). Therefore, the following page will work:

<%@ Page Language="C#" 
    Inherits="TestSite.DataPage`1[[System.String, mscorlib]]" %>

However, it’s not so simple. ASP.Net does not call Type.GetType to parse these strings; instead, it loops over every referenced assembly and calls Assembly.GetType on each one. This is why you don’t need to include the assembly name whenever using the Inherits attribute (which would have been necessary for Type.GetType) Ordinarily, this is very useful, but here, it comes back to bite you.
It is not possible to parse a type from one assembly with a generic parameter from a different assembly using Assembly.GetType, unless the generic parameter is in mscorlib.

Therefore, for example, it is not possible to create an ASPX page that inherits DataPage<DataLayer.Product> if DataLayer.Product is in a different assembly than DataPage. As a workaround, one can create a non-generic class which inherits DataPage<DataLayer.Product>, then make the ASPX page inherit this temporary class.

Next time: MVC magic

Building Connection Strings in .Net

Wed, 26 Jan 2011 14:20:00 -0800

.Net developers frequently need to build connection strings, especially when connecting to Access or Excel files using OleDB.
Code like the following has been written countless times:

//Bad code! Do not use!
string conn = "Data Source=" + openFileDialog1.FileName + "; "
            + "Provider=Microsoft.Jet.OLEDB.4.0;"
            + "Extended Properties=\"Excel 8.0\"";

This code looks innocuous at first glance, but will not work for all filenames. If the filename contains characters like ', ", ;, or =, this code will create an invalid connection string and throw an exception.

The correct way to build connection strings is to use one of the DbConnectionStringBuilder classes. This class implements a dictionary of key-value pairs in the connection string. It has a ConnectionString property which assembles the instance’s contents into a usable connection string. Unlike the string concatenation shown above, this property will correctly escape all values.

In addition, each of the database clients included with the .Net framework (SQL, OleDB, ODBC, Oracle, and Entity Framework) have their own inherited ConnectionStringBuilder classes in their respective namespaces. These classes add type-safe properties for the the keys supported by their databases, and handle any special cases when generating the connection string.

Thus, the correct way to write the above code is:

var connBuilder = new OleDbConnectionStringBuilder {
    DataSource = openFileDialog1.FileName,
    Provider = "Microsoft.Jet.OLEDB.4.0"
};
connBuilder["Extended Properties"] = "Excel 12.0 Macro";

As an added bonus, these classes implement ICustomTypeDescriptor, so they can be bound to a PropertyGrid to allow the end-user to edit the connection string. This can be seen in certain places in Visual Studio.

Don’t call Html.Encode in Razor Pages

Fri, 21 Jan 2011 01:35:00 -0800

One of the unique features of ASP.Net WebPages (formerly Razor) is automatic HTML encoding. All strings printed by embedded code nuggets (@ blocks) are automatically HTML-encoded.

In addition to this feature, Razor also includes the Html.Encode method, probably copied from ASP.Net MVC. Calling this method naively leads to a nasty surprise – the string will be double-encoded!
To see why, look more closely at a typical call: @Html.Encode("<text>"). This Razor markup will call Html.Encode, which returns the string "<text>". Since it returns a string and not an IHtmlString, the Razor engine will encode it again, and render &lt;text&gt;.

Careful thought indicates that this behavior is probably correct. The programmer (hopefully) knows that Razor will escape its output, so the call to Html.Encode should be an attempt to display encoded text. In fact, this is the simplest way to display HTML-encoded text in a Razor view.

However, even if it is correct, the behavior is unexpected and should not be relied upon. The unambiguous way to display encoded text is to call Html.Raw:

@Html.Raw(Html.Encode(Html.Encode("Double-encoded <html> text!")))

Although it is long and clunky, this clearly shows that the text will be double-encoded.

Exercise for the reader: Why is it also necessary to call Html.Raw?

Optional Parameters in C# < 4

Wed, 19 Jan 2011 22:04:00 -0800

C# 4.0 adds support for optional parameters.

The following code prints 4:

static void Main() {
    TestMethod();
}
static void TestMethod(int i = 4) {
    Console.WriteLine(i);
}

Optional parameters are a compiler feature. The compiler will emit a normal method with the IL [opt] attribute and a .param declaration that includes a default value:

.method hidebysig static void 
        TestMethod([opt] int32 i) cil managed
{
    .param [1] = int32(4)
    .maxstack 8
    L_0001: ldarg.0 
    L_0002: call void [mscorlib]System.Console::WriteLine(int32)
    L_0007: ret 
}

Earlier versions of the C# compiler will ignore this metadata. Therefore, you can call such methods in earlier versions of C#, but you will always need to pass the optional parameters.

You can also create methods with optional parameters in earlier versions of C#. You can force the compiler to emit this metadata using the [Optional] and [DefaultParameterValue] attributes.
(in the System.Runtime.InteropServices namespace)

The following C# 1 code will compile identically to the above:

static void TestMethod(
    [Optional, DefaultParameterValue(4)] int i
) {
    Console.WriteLine(i);
}

Since the older compilers cannot consume optional parameters, you will always need to pass i when calling this method from C# < 4. However, in C# 4, you can call this method without passing the parameter.

Unfortunately, the syntax parser used by VS2010 doesn’t recognize these attributes. Therefore, if you attempt to call a method using these attributes, inside the project that defines the method, without specifying the parameter, the IDE will show a syntax error. The compiler itself, however, will work correctly. I reported this bug on Connect.

The [DefaultParameterValue] attribute is optional; omitting it will use the type’s default value (0, an empty struct, or null, as appropriate) as the default.

These attributes can also be used to create methods with optional parameters using CodeDOM, which cannot emit the new syntax.

Writing output in Razor helpers using code

Wed, 12 Jan 2011 20:22:00 -0800

The new ASP.Net WebPages view engine (formerly Razor) allows you to create reusable parameterized blocks of HTML called helpers.

For example:

@helper Fibonacci(int count) {
    int current = 1, prev = 0;
    for (int i = 0; i &lt; count; i++) {
        @:@current, 
        int t = current;
        current += prev;
        prev = t;
    }
}

This helper will write out the first count Fibonacci numbers. It can be called by writing @Fibonacci(30) in the page that defines the helper.

Using Razor syntax in this code looks strange. Razor syntax is designed to write HTML tags. Since I’m printing plain text, I need to use the @: escape (or the <text> tag) in order to output my text. This small bit of code looks confusing and can get lost inside larger blocks of server-side code.

Instead, I can use an undocumented hack. Razor helpers are implemented by compiling a lambda expression as an Action<TextWriter>. The lambda expression receives a TextWriter parameter named __razor_helper_writer. (You can see this by writing a Razor page with a compilation error and clicking Show Complete Compilation Source) There is nothing preventing me from using this parameter yourself. (it even shows up in IntelliSense!)

Therefore, I can rewrite the helper as follows:

@helper Fibonacci(int count) {
    int current = 1, prev = 0;
    for (int i = 0; i &lt; count; i++) {
        __razor_helper_writer.Write(current + &quot;, &quot;);
        int t = current;
        current += prev;
        prev = t;
    }
}

Remember to correctly escape your text by calling Html.Encode. Since this writes directly to the output stream, it doesn’t get the benefit of Razor’s automatic escaping.

Note: This relies on undocumented implementation details of the Razor compiler, and may change in future releases. I would not recommend doing this.
Instead, you can write a function:

@functions {
    string Fibonacci(int count) {
        var builder = new System.Text.StringBuilder();
        int current = 1, prev = 0;
        for (int i = 0; i &lt; count; i++) {
            builder.Append(current).Append(&quot;, &quot;);

            int t = current;
            current += prev;
            prev = t;
        }
        return builder.ToString();
    }
}

This can be called the same way as the helper. Instead of using the special helper support, the call will just print a string, the same you print any other string. Therefore, the function’s output will be HTML-escaped. To prevent that, you can change the function to return an HtmlString.

Binding to lists of DataRows

Mon, 10 Jan 2011 20:30:00 -0800

.Net DataTables can be very useful when writing data-driven applications. However, they have one limitation: There is no obvious way to databind a grid (or other control) to an arbitrary list of datarows from a table.
You can bind to an entire table directly by setting a DataSource to the DataTable itself, and you can bind to a subset of a table by creating a DataView with a filter.

In general, you cannot bind to an IEnumerable<T> (eg, a LINQ query); the databinding infrastructure can only handle an IList (non-generic) or an IListSource. This is true for any kind of datasource. Therefore, to bind to any LINQ query, you need to call .ToList(). (Or .ToArray())

However, when binding to a DataTable, you can’t even use a List<DataRow>. If you try, you’ll get four columns (RowError, RowState, Table, and HasErrors) and no useful information. This happens because the List<DataRow> doesn’t tell the databinding infrastructure about the special properties of the DataRows. To understand the problem, some background is necessary

Databinding is controlled by the ListBindingHelper and TypeDescriptor classes. When you bind to a list, the ListBindingHelper.GetListItemProperties method is called to get the columns in the list. If the list implements the ITypedList interface, its GetItemProperties method is called. Otherwise, it will use TypeDescriptor to get the properties of the first item in the list. (this uses reflection)

The DataView class (which DataTable also binds through, using IListSource) implements ITypedList and returns DataColumnPropertyDescriptors that expose the columns in the table. This is why you can bind to a DataView or DataTable and see columns. However, when you bind to a List<DataRow>, there is no ITypedList that can return the columns as properties. It therefore falls back on reflection and shows the physical properties of the DataRow class.

To solve this issue, you need to wrap the list in a DataView so that you can take advantage of its ITypedList implementation. You can do that using the AsDataView() method. This method is only available on the DataTable and EnumerableRowCollection<T> classes; it cannot be called on an arbitrary LINQ query. You can only get an EnumerableRowCollection<T> by calling special versions of the Cast, OrderBy, Where, and Select methods from a DataTable.

Therefore, you can databind to a simple LINQ query by calling AsDataView() on the query. To bind to a List<DataRow>, or to a more complicated query, you can use an ugly hack:

List<DataRow> list = ...;
grid.DataSource = table.AsEnumerable()
                       .Where(list.Contains)
                       .AsDataView();

The AsEnumerable() call is not needed for typed datasets.

You can also call CopyToDataTable(), which will works on an arbitrary IEnumerable<DataRow>. However, it makes deep copies of the rows, so it isn’t helpful if you want the user to update the data, or if you want the user to see changes made (in code) to the original datarows.

Partial Type Inference in .Net

Thu, 30 Dec 2010 01:58:00 -0800

When designing fluent APIs, one issue that comes up is partial type inference. If a method has two type parameters, there is no way to call it and only specify one of the type parameters (and leave the other inferred by the compiler)

For example, suppose we are creating a type-safe wrapper around a parameterized SqlCommand.
Ideally, it would be called like this:

using(DbConnection connection = ...) {
    var result = connection.ExecuteScalar<int>(
        "SELECT COUNT(*) FROM TableName WHERE Modified > someDate",
        new { someDate }
    );
}

Where the generic parameter specifies the return type.

In order to implement this efficiently, one would create methods at runtime which add DbParameters for each property in the anonymous type, and store them in a static generic class.
It would look something like this:

static class Extensions {
    public static void AddParams<TParam>(this IDbCommand command, 
                                         TParam parameters)
        where TParam : class { 
        if (parameters != null) 
            ParamAdders<TParam>.Adder(command, parameters);
    }
    static class ParamAdders<TParam> where TParam : class {
        public delegate void ParamAdder(IDbCommand command,
                                        TParam parameters);
        public static readonly ParamAdder Adder = CreateParamAdder();

        private static ParamAdder CreateParamAdder() { 
            return ...; 
        }
    }
}

However, that requires that the ExecuteScalar extension method take TParam as a generic parameter. Since anonymous types can only be passed as generic parameters via type inference, this makes it impossible to pass the return type as a generic parameter.

To fix this issue, we can split the generic parameters across two methods. We can change the extension method to take a single generic parameter, and return a generic class with a method that takes the other generic parameter.

For example:

static class BetterExtensions {
    public static SqlStatement<T> Sql<T>(this IDbConnection conn,
                                         string sqlText) { 
        return new SqlStatement<T>(conn, sqlText); 
    }
}
class SqlStatement<TReturn> : IHideObjectMembers {
    public SqlStatement(IDbConnection connection, string sql) { 
        Connection = connection; 
        Sql = sql;
    }
    public IDbConnection Connection { get; private set; }
    public string Sql { get; private set; }

    public TReturn Execute() { return Execute<object>(null); }
    public TReturn Execute<TParam>(TParam parameters)
         where TParam : class {
        return ...;
    }
}

I use an IHideObjectMembers interface to hide the methods inherited from Object from IntelliSense. Note that the interface must be defined in an assembly outside of your solution. (or, to be more precise, that isn’t a Project reference)

This version would be called like this:

using(IDbConnection connection = null) {
    var result = connection.Sql<int>(
        "SELECT COUNT(*) FROM TableName WHERE Modified > someDate"
    ).Execute(new { someDate });
}

The return type is specified explicitly in the call to Sql<T>(), and the parameter type is passed implicitly to Execute<TParam>().

Simplifying Value Comparison Semantics

Wed, 29 Dec 2010 03:10:00 -0800

A common chore in developing real-world C# applications is implementing value semantics for equality. This involves implementing IEquatable<T>, overriding Equals() and GetHashCode(), and overloading the == and != operators.

Implementing these methods is a time-consuming and repetitive task, and is easy to get wrong, especially GetHashCode(). In particular, the best way implement GetHashCode() is much more complicated than return x.GetHashCode() ^ y.GetHashCode().

To simplify this task, I created a ValueComparer class:

///<summary>
/// Contains all of the properties of a class that 
/// are used to provide value semantics.
///</summary>
///<remarks>
/// You can create a static readonly ValueComparer for your class,
/// then call into it from Equals, GetHashCode, and CompareTo.
///</remarks>
class ValueComparer<T> : IComparer<T>, IEqualityComparer<T> {
    public ValueComparer(params Func<T, object>[] props) {
        Properties = new ReadOnlyCollection<Func<T, object>>(props);
    }

    public ReadOnlyCollection<Func<T, object>> Properties
            { get; private set; }

    public bool Equals(T x, T y) {
        if (ReferenceEquals(x, y)) return true;
        if (x == null || y == null) return false;
        //Object.Equals handles strings and nulls correctly
        return Properties.All(f => Equals(f(x), f(y)));    
    }

    //http://stackoverflow.com/questions/263400/263416#263416
    public int GetHashCode(T obj) {
        if (obj == null) return -42;
        unchecked {
            int hash = 17;
            foreach (var prop in Properties) {
                object value = prop(obj);
                if (value == null)
                    hash = hash * 23 - 1;
                else
                    hash = hash * 23 + value.GetHashCode();
            }
            return hash;
        }
    }

    public int Compare(T x, T y) {
        foreach (var prop in Properties) {
            //The properties can be any type including null.
            var comp = Comparer.DefaultInvariant
                .Compare(prop(x), prop(y));    
            if (comp != 0)
                return comp;
        }
        return 0;
    }
}

This class implements an external comparer that compares two instances by an ordered list of properties.

ValueComparer can be used as a standalone IComparer<T> or IEqualityComparer<T> implementation.

It can also be used to implement value semantics within a type.
For example:

class Person : IComparable<Person>, IEquatable<Person>, IComparable {
    public string FirstName { get; set; }
    public string LastName { get; set; }
    public string Address { get; set; }
    public string Phone { get; set; }
    public string Email { get; set; }

    public override int GetHashCode() { return Comparer.GetHashCode(this); }
    public int CompareTo(Person obj) { return Comparer.Compare(this, obj); }
    int IComparable.CompareTo(object obj) { return CompareTo(obj as Person); }
    public bool Equals(Person obj) { return Comparer.Equals(this, obj); }
    public override bool Equals(object obj) { return Equals(obj as Person); }
    static readonly ValueComparer<Person> Comparer = new ValueComparer<Person>(
        o => o.LastName,
        o => o.FirstName,
        o => o.Address,
        o => o.Phone,
        o => o.Email
    );
}

To simplify this task, I created a code snippet:

<?xml version="1.0" encoding="utf-8" ?>
<CodeSnippets  xmlns="http://schemas.microsoft.com/VisualStudio/2005/CodeSnippet">
    <CodeSnippet Format="1.0.0">
        <Header>
            <Title>ValueComparer</Title>
            <Shortcut>vc</Shortcut>
            <Description>Code snippet for equality methods using ValueComparer</Description>
            <Author>SLaks</Author>
            <SnippetTypes>
                <SnippetType>Expansion</SnippetType>
            </SnippetTypes>
        </Header>
        <Snippet>
            <Declarations>
                <Literal Editable="false">
                    <ID>classname</ID>
                    <ToolTip>Class name</ToolTip>
                    <Default>ClassNamePlaceholder</Default>
                    <Function>ClassName()</Function>
                </Literal>
            </Declarations>
            <Code Language="csharp">
                <![CDATA[public override int GetHashCode() { return Comparer.GetHashCode(this); }
        public int CompareTo($classname$ obj) { return Comparer.Compare(this, obj); }
        int IComparable.CompareTo(object obj) { return CompareTo(obj as $classname$); }
        public bool Equals($classname$ obj) { return Comparer.Equals(this, obj); }
        public override bool Equals(object obj) { return Equals(obj as $classname$); }
        static readonly ValueComparer<$classname$> Comparer = new ValueComparer<$classname$>(
            o => o.$end$
        );]]>
            </Code>
        </Snippet>
    </CodeSnippet>
</CodeSnippets>

It can also be downloaded here; save it to My Documents\Visual Studio 2010\Code Snippets\Visual C#\My Code Snippets\

When shouldn’t you write ref this?

Wed, 22 Dec 2010 22:54:00 -0800

Last time, we saw that the this parameter to an instance method in a struct is passed by reference, allowing the method to re-assign this or pass it as a ref parameter.

Due to limitations in the CLR, the this parameter to an iterator method is not a reference to the caller’s struct, and is instead a copy of the value. Quoting the spec (§7.6.7)

When this is used in a primary-expression within an instance method or instance accessor of a struct, it is classified as a variable. The type of the variable is the instance type (§10.3.1) of the struct within which the usage occurs.

If the method or accessor is not an iterator (§10.14), the this variable represents the struct for which the method or accessor was invoked, and behaves exactly the same as a ref parameter of the struct type.

If the method or accessor is an iterator, the this variable represents a copy of the struct for which the method or accessor was invoked, and behaves exactly the same as a value parameter of the struct type.

To explain why, you’ll need to understand how iterators are compiled. As Jon Skeet explains, an iterator method is compiled into a nested class that implements IEnumerable, with the original code transformed into a state machine. This nested class has fields to store the method’s parameters (which includes this for instance methods) so that the iterator code can use them.

This is why iterators cannot take ref parameters; the CLR cannot store a reference (as opposed to a reference type) in a field. Therefore, the this parameter is passed to iterator methods by value, not by reference.
This is also why anonymous methods in structs cannot use this; anonymous methods are also compiled to methods in separate classes and so cannot inherit the ref parameter.

This means that if you change the Mutate method from before into an iterator, the code will still compile (!), but the calling method will not see the changes.

static void Main() {
    Mutable m = new Mutable();
    m.MutateWrong().ToArray();    //Force the iterator to execute
    Console.WriteLine("In Main(): " + m.Value);
}

struct Mutable {
    public int Value;
    
    public IEnumerable<int> MutateWrong() {
        this = new Mutable();
        MutateStruct(ref this); 
        Console.WriteLine("Inside MutateWrong(): " + Value);
        yield break;
    }
}
static void MutateStruct(ref Mutable m) { 
    m.Value++; 
}

This code prints

Inside MutateWrong(): 1
In Main(): 0

In summary, don’t mutate structs in iterators (or at all, if you can help it).
I don’t know why this isn’t a compiler error.

When can you write ref this?

Wed, 22 Dec 2010 20:50:00 -0800

Usually, you cannot pass ref this as a parameter, since this is not a writable field. However, that’s not true for value types. The this field of a value type is a writable value.

To quote the spec (§5.1.5)

Within an instance method or instance accessor of a struct type, the this keyword behaves exactly as a reference parameter of the struct type (§7.6.7).

Therefore, the following code prints 1:

static void Main() {
    Mutable m = new Mutable();
    m.Mutate();
    Console.WriteLine(m.Value);
}

struct Mutable {
    public int Value;
    
    public void Mutate() {
        this = new Mutable(); 
        MutateStruct(ref this); 
    }
}
static void MutateStruct(ref Mutable m) { 
    m.Value++; 
}

In practice, this should never come up, since mutable structs are evil and should be avoided at all costs.

Next time: This doesn’t always work.

Nothing vs Null

Wed, 22 Dec 2010 17:28:00 -0800

VB.Net’s Nothing keyword is is not the same as C#’s null. MSDN states, “Assigning Nothing to a variable sets it to the default value for its declared type. If that type contains variable members, they are all set to their default value”.

In other words, the Nothing keyword is actually equivalent to C#’s default(T) keyword, where T is the type that the expression is used as.

This can lead to nasty surprises with nullable types in conditional operators.
In C#, the expression (...) ? null : 1 will not compile, since “there is no implicit conversion between '<null>' and 'int'”. Since null is an untyped expression, the type of the conditional is inferred to be int, resulting in an error because null cannot be converted to int.

In VB.Net, by contrast, the equivalent expression, If((...), Nothing, 1), will compile, but will have unexpected results. Here too, Nothing is an untyped expression, so the type of the conditional is inferred to be Integer. However, unlike null, Nothing can be converted to Integer, so this is compiled as If((...), 0, 1), which is probably not what the programmer intended.

In both languages, the solution is to use an expression which is actually typed as int?, by writing new int?(), (in C#) or New Integer?() (in VB.Net).

Animating Table Rows with jQuery

Tue, 21 Dec 2010 17:19:00 -0800

jQuery contains a powerful and flexible animation engine. However, it has some limitations, primarily due to underlying limitations of CSS-based layout

For example, there is no simple way to slideUp() a table row (<tr> element). The slideUp animation will animate the element’s height to zero. However, a table row is always tall enough to show its elements, so the animation cannot actually shrink the element.

To work around this, we can wrap the contents of each cell in a <div> element, then slideUp() the <div> elements. Doing this in the HTML would create ugly and non-semantic markup, so we can do it in jQuery instead.

For example: Demo

$('tr')
    .children('td, th')
    .animate({ padding: 0 })
    .wrapInner('<div />')
    .children()
    .slideUp(function() { $(this).closest('tr').remove(); });

Explanation:

Get all of the cells in the row
Animate away any padding in the cells
Wrap all of the contents of each cell in one <div> element for each cell (calling wrapInner())
Select the new <div> elements
Slide up the <div>s, and remove the rows when finished.

If you don’t remove the rows, their borders will still be visible. Therefore, if you want the rows to stay after the animation, call hide() instead of remove().

Requiring Inherited Types in Generic Constraints

Fri, 17 Dec 2010 17:08:00 -0800

A generic class can specify that its generic parameter must inherit a type. However, there is no obvious way in general to prevent clients from passing the base type itself.

For example, take the following set of types:

abstract class Entity  { }

class Person : Entity { }
class Boat : Entity { }
class Car : Entity { }

class Repository<TEntity> where TEntity : Entity { }

This allows the type Repository<Entity>, which doesn’t make logical sense.

In this particular case, we could prevent that by changing the generic constraint to where TEntity : Entity, new(). Since the base Entity class is abstract, that would disallow a Repository<Entity>. However,if the concrete entities also don’t have default constructors, this wouldn’t work. Similarly, had the base type been an interface, we could add a : class constraint.

There is an (somewhat) ugly hack that can be used to prevent parameterizations of the base class in arbitrary cases (as long as you control every type involved):

abstract class Entity  { }
interface IConcreteEntity { }    //Marker interface

class Person : Entity, IConcreteEntity { }
class Boat : Entity, IConcreteEntity { }
class Car : Entity, IConcreteEntity { }

class Repository<TEntity> 
      where TEntity : Entity, IConcreteEntity { }

Specifically, we can add a marker interface which is implemented by all concrete implementations of the base type. We can then constrain a generic parameter to inherit both the base type and the marker interface. Since the base class itself does implement the marker interface, it will not be valid as a parameter.
Note that the marker interface must be implemented (perhaps indirectly) by every single concrete implementation.

Nested Iterators, part 2

Thu, 16 Dec 2010 21:50:00 -0800

In part 1, we discussed the simple approach to making a nested iterator. However, we fell short of a completely lazy nested iterator.

In simple cases, we can make an separate iterator method for the subsequence:

IEnumerable<IEnumerable<int>> FullyLazy() {
    for(int i = 0; i < 10; i++) 
        yield return Inner(i);
}
IEnumerable<int> Inner(int i) {
    for(int j = 0; j < 10; j++)
        yield return i * 10 + j;
}

Note that this is actually smaller than the single-method implementation!

This seems to work very well; the inner iterator code for a particular subsequence will not execute at all unless that subsequence is actually enumerated.

However, this approach falls short in practice. To see why, consider a real-world example.
Here is a fully lazy implementation of a Partition method, which converts a sequence into a 2D “jagged IEnumerable” where each subsequence (partition) contains n elements.

class Ref<T> { 
    public Ref(T value) { Value = value; } 
    public T Value { get; set; } 
}

public static IEnumerable<IEnumerable<T>> Partition<T>
              (this IEnumerable<T> sequence, int size) {
    using(var enumerator = sequence.GetEnumerator()) {
        var isFinished = new Ref<bool>(false);
        while(!isFinished.Value) {
            yield return PartitionInner(
                 enumerator, size, isFinished
            );
        }
    }
}
static IEnumerable<T> PartitionInner<T>
      (IEnumerator<T> enumerator, int size, Ref<bool> isFinished) {
    while(size --> 0) {
          isFinished.Value = !enumerator.MoveNext();
        if (isFinished.Value) yield break;
        yield return enumerator.Current;
    }
}

This method is complicated by the need to communicate back from the inner iterator to the outer one. Since iterators cannot have ref parameters, I need to make a “box” class that holds a reference to an int. This allows the outer iterator to find out when the sequence finishes.

In addition, this implementation has a subtle bug: If the last partition is full, it will return an extra, empty, partition after it.
Fixing this issue would require that the outer method call MoveNext() before each call to the inner method. This makes the code even more complicated; I won’t list it here (unless people really want me to)

This design has a more fundamental problem: The behavior of the outer method is determined by the inner ones. It will only work if each inner IEnumerable<T> is enumerated exactly once, before the next MoveNext() call to the outer iterator. If, for example, you iterate each inner iterator twice, it will behave unexpectedly:

Enumerable.Range(1, 8)
          .Partition(3)
          .Select(p => String.Join(", ", p.Concat(p)));

This code is intended to create strings that contain each partition twice. It is supposed to return

1, 2, 3, 1, 2, 3
4, 5, 6, 4, 5, 6
7, 8, 7, 8

However, it actually creates the strings

1, 2, 3, 4, 5, 6
7, 8

Enumerating the inner iterator a second time will end up consuming extra items from the original sequence.

Thus, in order to produce enumerables that behave correctly, the inner enumerator needs to cache its items; it is impossible to make a fully lazy Partition method.

It gets worse. In order to allow callers to write thingy.Partition(5).Skip(2), (which should skip the first two partitions) the outer enumerator needs to cache all items, because it cannot assume that the inner iterators will be called at all.

Thus, the laziest possible Partition method must use the original approach:

public static IEnumerable<IEnumerable<T>> PartitionCorrect<T>
              (this IEnumerable<T> sequence, int size) {
    List<T> partition = new List<T>();
    foreach(var item in sequence) {
        partition.Add(item);
        if (partition.Count == size) {
            yield return partition;
            partition = new List<T>();
        }
    }
    if (partition.Count > 0)
        yield return partition;
}

It would be possible to write a slightly lazier version that combines these two approaches and passes a List<T> to the inner iterator, which would return whatever is in the list, then enumerate if necessary to fill the list. However, I’m too lazy to do it. (unless people really want me to)

Nested Iterators, part 1

Thu, 16 Dec 2010 16:26:00 -0800

C# 2.0 introduced a powerful feature called an iterator, a method which returns an IEnumerable<T> or IEnumerator<T> using the new yield keyword.

Using an iterator, you can quickly and easily create a method which returns lazily a sequence of values. However, lazily returning a sequence of sequences (IEnumerable<IEnumerable<T>>) is not so simple.

The obvious approach is to yield return a List<T>:

IEnumerable<IEnumerable<int>> SemiLazy() {
    for(int i = 0; i < 10; i++) {
        List<int> numbers = new List<int>();
        for(int j = 0; j < 10; j++)
            numbers.Add(i * 10 + j);
            
        yield return numbers;
    }
}

(This can be shortened to a single LINQ statement, but that’s beyond the point of this post: Enumerable.Range(0, 10).Select(i => Enumerable.Range(10 * i, 10)) )

This approach is very simple, but isn’t very lazy; each subsequence will be computed in its entirety, whether it’s consumed or not.

This approach also has a subtle catch: the iterator must return a different List<T> instance every time.

If you “optimize” it to re-use the instance, you’ll break callers which don’t use the subsequences immediately:

IEnumerable<IEnumerable<int>> Wrong() {
    List<int> numbers = new List<int>();
    for(int i = 0; i < 10; i++) {
        numbers.Clear();
        for(int j = 0; j < 10; j++)
            numbers.Add(i * 10 + j);
            
        yield return numbers;
    }
}

Calling SemiLazy().ToArray()[0].First() will return 0 (the first element in the first subsequence); calling Wrong().ToArray()[0].First() will return 90 (since all subsequences refer to the same instance).

Next: How can we achieve full laziness?

On copy prevention in HTML, part 3

Fri, 27 Apr 2007 20:56:00 -0700

My previous post stretched the limit of simple copy prevention. Beyond this point, it gets very complicated. Before continuing, some thought is in order. Who are you trying to prevent from copying your text? Why shouldn't the text be copied? Unless you are trying to stop a hardcore developer, the previous methods should suffice. Also, what kind of copying are you trying to prevent? If you are trying to prevent the copier from copying into a web page, it is significantly harder, because he can copy your source and it will display normally.

I can think of two ways to prevent the copier from using a screenreader to copy your text.

Put all of the text into a single, CAPTCHA-like image. This way, the screenreader will not be able to read the text. However, this will also make it more difficult for legitimate people to read your text. Also, the copier could simply insert the large image as-is into his document. This risk could be mitigated by watermarking it with your name.
Break apart the text into many different images, each one somewhat smaller than a letter. This could be done in server-side code. Then, use a JavaScript timer to alternate the images so that the screenreader will never see all of the text at once. To prevent the copier from modifying the JavaScript to show all of the images, alternate them with other images. For example, write a server-side script that takes X and Y coordinates, and a timer index. This script would either a white image, or the chunk of text at the given coordinates, depending on the timer index. To prevent the copier from using GDI to OR-blit screenshots from different times, (or from using Photoshop to make the white transparent, then pasting them together) the white could have random black patterns. However, this will make the text hard to read. If a JavaScript timer isn't fast enough to make the text legible, it could be done in Flash.

Preventing the copier from copying your content into HTML is more difficult. No matter how obscure your source is, the copier could use a tool like Firebug to copy your DOM source using the innerHTML property. Here too, there are several options.

Use my Scrambler (see part 2), and put your name anywhere within the scrambled text. (for example, you could put in, in the middle, Written by Your Name Here; do not copy) It is virtually impossible for the copier to extricate the spans that form your name and then fill the resulting gap. This could also be done in an image. However, if you put your name at the end, the copier could position a white DIV to hide it. Or, if his name is similar in length, he could position a white DIV with his name to hide it.
If you are only worried about part of your text, scramble the entire page. It would be extremely difficult for the copier to extract the sensitive part. For example, you could add a lengthy copyright header before the content, and scramble it with the content. However, the copier could position the containing DIV so that the copyright header is above the top of the DIV.
Break the text into a large set of images, and use client-side JavaScript to execute a server-generated script that adds these images from another server-side script. For every request, require a single-use authorization token returned by the previous request. The initial page request would include the auth-token for the first script, and each image would be preceded by an AJAX request for auth-tokens for the image and for the next AJAX request. The first script would have an auth-token for the first AJAX request embedded within it. To make it more difficult for an attacker to get an auth-token from the initial script, you could encode the script before sending it, and decode on the client, then pass it to the eval() function. All responses would have the no-cache header to prevent the copier from taking the images out of the cache, and the images would be used by the background-image attribute on a DIV to prevent Save Image As (only necessary if Save Image As doesn't redownload the image from the server). The server would track auth-tokens in a database, and delete them when used. If you do all this, the only way the copier could get the images from the page would be Print Screen. To prevent that, make the images flicker as described earlier. The copier could, however, load your page with JavaScript disabled, download and decode the script, convert it to a full programming language (eg, C#), and use the script's embedded auth-token to send the "AJAX" requests over HTTP (for example, using .NET's HttpWebRequest class) and download the images. To prevent this, make the auth-tokens expire after about 30 seconds. If any auth-token is expired, all subsequent images should form something different. (maybe Service Unavailable, or random black pixels, or a different text) By the time the copier finishes writing his program, his "stolen" auth-token will have expired, and he will not know what he did wrong. To prevent him from trying again, you could blacklist his IP address after receiving an expired request, and embed IP address in auth-tokens. You could also set and require some innocuous-seeing cookies on the server for every request. The copier wouldn't notice these cookies, and when he requests the images without these cookies in his request, you could send him whatever you want. Please note that if your page requires a login, the copier probably will check cookies. And, if the copier is being paid by the hour (this is quite likely; otherwise, he'd give up), he might even thank you for doing all this.

On copy prevention in HTML, part 2

Mon, 16 Apr 2007 21:49:00 -0700

The methods discussed in my previous post are crude and ugly. Most of the time, they do work, but they do nothing to prevent the user from viewing the source and copying the text from there. Also, the user has a right to select text that should not be denied. For example, if one wants to show someone part of a large document, the easiest way to do that is to select the part.

ZSkTuKpBrLljyVW GmtoBbO MVocxRvoopy zKYtQahiDEsh LLtQexowSEtDnIg. NoyticDMe thiMaDVnZt, whGZenjEE ufapdeIPasBZxtgCeYWDd, iSt BlMNooks KPzRlkeeGifkshqdheodB tIVnoMNtEal nySouQnAqVsensegX. cUHHoNweqdcvecFGrU,PGZ pMibt rqcanrbKkn eHstilqTulOE beRPuv STwaQsyTePXvplRoCectxeKAjd jVpnXljoDYrDlrmaKlly.B IiLwokzofk at itsjw JXsCoIuuFhjrce:

<SPAN style="position: static;left:-9477px;position: absolute;">Z</SPAN>
<SPAN style="position: static;left:-9765px;position: absolute;">S</SPAN>
<SPAN style="position: static;left:-9586px;position: absolute;">k</SPAN>
<SPAN style="position: static;left:-9373px;position: absolutee;">T</SPAN>
<SPAN style="position: static;left:-9734px;position: absolute;">u</SPAN>
<SPAN style="position: static;left:-9872px;position: absolute;">K</SPAN>
<SPAN style="position: static;left:-9773px;position: absolute;">p</SPAN>
<SPAN style="position: static;left:-9326px;position: absolute;">B</SPAN>
<SPAN style="position: static;left:-9195px;position: absolutee;">r</SPAN>
<SPAN style="position: static;left:-9413px;position: absolute;">L</SPAN>
<SPAN style="position: static;left:-9196px;position: absolute;">l</SPAN>
<SPAN style="position: static;left:-9737px;position: absolute;">j</SPAN>
<SPAN style="position: static;left:-9897px;position: absolutee;">y</SPAN>
<SPAN style="position: static;left:-9014px;position: absolute;">V</SPAN>
<SPAN style="position: static;left:-9893px;position: absolute;">W</SPAN>
<SPAN style="position: static;left:-9103px;position: absolutee;"> </SPAN>
...

gicotxaoyenc:yehosrk,mutaryslts.tsitatkiisuoptcrebhttnihdoLfI

<span style="position: absolute; left: 241px; top: 0px;">g</span>
<span style="position: absolute; left: 263px; top: 0px;">i</span>
<span style="position: absolute; left: 117px; top: 0px;">c</span>
<span style="position: absolute; left: 17px; top: 20px;">o</span>
<span style="position: absolute; left: 276px; top: 0px;">t</span>
<span style="position: absolute; left: 287px; top: 0px;">x</span>
<span style="position: absolute; left: 129px; top: 0px;">a</span>
<span style="position: absolute; left: 9px; top: 20px;">o</span>
<span style="position: absolute; left: 13px; top: 0px;">y</span>
...

These texts were generated by a script (currently unavailable). It can do two things: Inflate (the first paragraph above) and Scramble (the second paragraph).

Using the Inflate option will add random characters in SPANs positioned absolutely between nine and ten thousand pixels to the left. When the user selects text, the random characters will also be selected, and when the text is copied, they will also be copied. It will randomly add up to five letters between every two characters, and it will ignore text in TEXTAREAs, SCRIPTs, and SELECTs. Such text could be "deflated" by copying the source, then removing all text that matches the regular expression /<span[a-z0-9 ;:="']*?>[A-Za-z]</span>/ , removing all SPAN tags that contain a letter and an attribute. Therefore, I put each original letter in a very similar SPAN tag, complete with a random location, and gave both types of spans position:absolute and position:static, in different orders. This could also be matched by a regular expression, but it would be much more complicated, and I will not list it here. It would also be possible to write a GreaseMonkey script that would loop through the SPANs and delete all of them which has a position attribute equal to absolute. However, it would probably be easier to retype it manually.

Using the Scramble option will put each letter in to a SPAN, position them absolutely at their correct location, and randomize the order. Therefore, when the user selects the text, the selection will be scrambled, and when it is pasted, it will show up as nonsense. Spaces are rendered pointless by the procedure, and are therefore removed. Scrambled text will not select cleanly, but is nearly impossible to descramble. It would be possible to write a GreaseMonkey script that would sort the SPANs by top, then by left, but it would be much easier to retype the text manually.

When using this approach, remember to position the text's container, or it will show up in unexpected places. In addition, this approach will completely break word wrap,and must therefore be placed in a container with a fixed width. This width should be entered in the Width textbox in the scrambler so that the text will flow correctly.

These methods will definitely prevent all but the most determined and technically skilled copiers. However, they do not prevent OCR screenreaders. This will be discussed in part 3.

On copy prevention in HTML, part 1

Sun, 08 Apr 2007 18:27:00 -0700

Many web developers like to prevent their viewers from copying their text. While I do not approve of this, there are cases where it is appropriate.

The simplest way to achieve this is to use the IE only attribute UNSELECTABLE and the FireFox only css style -moz-user-select. Such HTML looks like this:

<DIV unselectable="on"
style="-moz-user-select:none;">
You can't select me.
</DIV>

You can't select me.

To make the HTML and CSS validate, one could do this in Javascript: Elem.unselectable = "on"; Elem.style.MozUserSelect = "none";

However, this method only works in IE and Firefox. In addition, in IE, it doesn't work very well, and if a user tries hard, he will end up selecting the text.

A slightly better way to do it is to handle the onselectstart event (for IE) and the onmousedown event (for everything else) and return false. This will prevent the browser from handling the events. This results in something like this:

<DIV
onselectstart="return false;"
onmousedown="return false;" >
You can't select me.
</DIV>

You can't select me.

The problem with these methods is that they do nothing to prevent a user from reading the HTML source. This is discussed in the next part.