SLaks.Blog

Code Snippets: Removable = Tokens

2016-12-26T00:00:00+00:00

Or, torturing compilers for fun ~~and profit~~.

I recently tweeted an interesting C# challenge:

C# Quiz: Construct a valid program which remains valid when a single = token is removed.
— Schabse Laks (@Schabse) December 21, 2016

This has enough solutions to warrant a blog post, explaining how each answer works.

Note that you must remove an = character which is parsed as a single token; that means you can’t remove an = from a comment (which is not a token at all), or one which is merely part of a larger token like >=, =>, ==, or !=.

The = token is valid in 2 places: Assignments and variable (or similar) initializers. We must construct situations where the 2 tokens surrounding the = token remain valid in juxtaposition once it’s removed. Since there aren’t many places allow two identifiers in sequence, this is not easy.

First, there are 3 simple answers:

1: The `Color Color` trick

int Int32;
void M() {
    Int32 = Int32;
}

This utilizes the Color Color case; if an identifier can refer to either a typename or a simple name in the same scope, it’s treated as both, and can be used as either meaning.

With the = sign, this is an assignment, and both Int32s refer to the field. Without the = sign, it’s a variable declaration; the first Int32 refers to the System.Int32 type, and the second is a variable name (hiding the field).

Variant: Casting

Using the same Int32 declaration as above, you can also construct a cast:

int x = (Int32) = 0;

This works because of what appears to be a bug in the compiler; the left side of an assignment expression can be wrapped in parentheses. Without the =, this becomes a cast.

Since the cast alone is not valid as a statement, I need to wrap it in an assignment. This also means the Int32 variable must be of a type convertible to the Int32 type (since both are assigned to x here); the previous answer also works with string Int32 or any other type.

2: Collapsing 2 tokens to 1.

Credit to Josh Varty for this solution

int x=3;

This is a clever cheat; instead of merely removing a token, it utilizes the missing whitespace to combine the surrounding tokens into a single name.

3. Invocation expression

Action<int?> a = null;
a = (null);

This trick works because variables of delegate type are both callable and (unlike functions) assignable. By wrapping the right side of the assignment in parentheses, this becomes a valid function call when the = is removed.

Making this compile is a bit tricky; the right side must be convertible to both the delegate type and its parameter. null is the simple way to meet that requirement (and will come up again in more-complex answers). To avoid the null, you must declare your own delegate that accepts itself as a parameter; this is impossible to achieve with the built-in Action<T> delegates (since you can’t write Action<Action<Action<...>>> forever). example

Variant: `ref` return

ref Action<int?> M() => ref M();
void M2() {
    M () = (null);
}

I can also do this using another new C# 7 feature, ref returns, which allow a parenthesized expression on the left hand side of an assignment.

This answer calls M() whether the = token exists or not. With an =, it assigns null to its return value (by reference). Without the =, it calls that return value, passing null as a parameter.

The remaining answers are much more complicated, relying on clever ways to combine or abuse language features new to C# 6 and 7.

4. Lambda expressions vs. Expression-bodied methods

Func<int?> C = () => null;

This uses the = sign to switch between two entirely different declaration syntaxes; variables and expression-bodied methods.

With the =, this is a class field of type Func<int?>, initialized to a simple lambda expression. Removing the = changes it to a method whose return type is Func<int?>, with an expression body that returns null.

Like the previous example, I use null to be convertible to both the delegate type itself (for the field initializer) and its return value (for the expression-bodied method); I could also have used a delegate D D(); and returned C itself.

5. Splitting a class member in half

class async { 
    Func<int, Task<int?>> A { get; } = async(int B) => new int?();
}

Here’s where the real evil comes in to play. C# 6 introduces auto property initializers. This is the first time that the assignment operator can follow something other than a name (or array access), giving me new horizons to explore in its removal.

In particular, the assignment here is optional, and, when it’s omitted, no semicolon is needed to terminate the expression. This opens up new opportunities that aren’t possible in methods because of those pesky semicolons. All I need to do is create an initializer expression that becomes a valid class member declaration once the = is removed.

To do this, I turn to the powers of contextual keywords. When parsed as an expression, the right hand side of this initializer is an async lambda expression. Once I remove the =, it becomes an expression-bodied constructor (new to C #7) for a class named async. The body after the => must be valid as both a lambda that returns the type declared for A and as a statement (since constructors have no return value), so => null won’t work. The simplest expression which is valid as a statement and returns a value is new blah(), so I use that.

Would-be variant:

class async {
    Func<int, Task<int?>> A { get; } = async B => null;
}

In principle, I should be able to replace the expression-bodied constructor with a simple expression-bodied property, declaring a property of type async. Here, the body after the => is a property getter, so I can return null like I usually do.

However, due to a bug in the parser, expression-bodied properties cannot have a type named async, so this doesn’t actually compile.

6. Tuple Deconstruction

Action<int, int> var(int a, int b) => null;
int x, y;
void M() {
    var (x, y) = (1, 2);
}

This is similar to #3 and its variant, taking advantage of another new C# 7 feature, tuple support. Like the previous answer, this adds new possibilities to the = operator; the left-hand side can now be a parenthesized list. Without the =, the parenthesized list is not legal as-is, so we need to turn it into a parameter list for a method call.

With the =, this is a combination of a tuple literal and a deconstructing variable declaration. Removing the = completely rewrites the parse tree, turning it into a chained call to a function named var with two parameters, followed by a call to its returned delegate with two more parameters. x and y must be declared separately as fields so that they can be passed as parameters without = but be declared as variables with =.

You can also replace the tuple literal with an explicit tuple construction (demo), but the method must still be named var, since deconstructing assignment is only valid with var.

7. Local functions

Func<Task<int>> Int32;
Int32 = async() => 7;

This is a cleverer variant of #4, using the = to turn a method declaration into an assignment. This time, I combine expression-bodied methods with C# 7’s new local functions.

Local functions allow me to bring the expression-bodied method trick into statement context, allowing me to turn it into a statement rather than a declaration. This gives me more options to make it stay when adding the =.

This example starts as async lambda expression assigned to a variable named Int32 (which therefore must be of a delegate type that returns a Task). Removing the = turns this into a (non-async) local function named async that returns Int32. Like many of the other answers, this utilizes the Color Color trick to allow Int32 to be both a return type and an assignable name. Unlike #4, this doesn’t change the expression from a lambda to a return value, so I don’t need to return null to be compatible with both variants.

Bonus challenge: Construct a valid C# program with 2 = tokens that remains valid when either or both of them are removed.

Disclaimer: No compilers were (permanently) harmed in the writing of this blog post.

Common Cryptographic Pitfalls

2015-11-18T00:00:00+00:00

When writing code that deals with security or cryptography, there are a number of mistakes that many people make; some obvious and some quite subtle. This post describes the most common mistakes I’ve seen and why they’re wrong.

Don’t re-invent the car

Correctly using cryptographic primitives is hard. If at all possible, you should not use raw cryptographic primitives (even well-accepted ones like AES, RSA, or SHA2) directly; instead, you should use professionally-built and reviewed protocols that use these systems, such as TLS, NaCl, Keyczar, and others.

There are a variety of subtle issues that professional cryptographers know about and you don’t (such padding and timing vulnerabilities), and these higher-level wrappers address these issues for you.

There have been many security issues resulting from projects that attempt to build their own protocols, and get these wrong.

Don’t re-invent the wheel

Even if you do need to use primitives directly, stick to known, well-researched primitives, and use the best ones available.

In particular, please don’t try to build your own encryption or hash algorithms. Cryptography is exceptionally complicated; unless you actually have substantial experience in the mathematical side of cryptanalysis, you are very unlikely to create a secure algorithm. Standard algorithms like AES have had years of research, and yet still do not have known vulnerabilities (when used correctly).

On a similar note, try not to write your own implementations of standard algorithms; instead, stick to existing, hardened, well-tested codebases. These implementations have been hardened to avoid leaking data through timing issues with early termination or CPU caching (eg, not using secret information in CPU branching or loop boundaries).

Learn as much as possible

Especially if you do use primitives directly, learn about things like padding attacks, replay vulnerabilities, and the like. If you have time, take a course about cryptography (eg, Stanford’s online course). If you don’t have time for a full course, at least read the Wikipedia articles about algorithms you use.

Every bit of cryptographic lore you learn may help you catch a mistake in your design or implementation and stop a real-world attacker.

If you use cryptography in production, you should keep up-to-date with the latest developments; read cryptographic blogs or follow cryptographers on Twitter. Some good people to follow include Bruce Schneier and Matthew Green.

Don’t reuse keys

Don’t use the same key for two different purposes. In worst-case scenarios, you can accidentally serve an endpoint that decrypts input with the same key used elsewhere to encrypt things (yes, this has happened). Even if you use keys only with unrelated cryptosystems, you still risk leaking information that a cryptanalyst can use to derive information about the key.

Ideally, each cryptosystem you use, and each scenario you use it in, should have its own unique key, each generated by a secure random number generator. If, for whatever reason, you can’t afford that much randomness, you can accept a single random blob (at least as large as the largest key you need), and use HMAC hashes with different (hard-coded) HMAC keys to derive a separate key from that blob for each use.

Create a threat model

It is important to understand what exactly you’re trying to prevent, and who you’re trying to block. Are you just trying to prevent attackers from impersonating users (eg, authentication for a public forum)? Are you trying to prevent attackers from reading messages at all (eg, personal chat)? Are you trying to prevent attackers from knowing who your users are and who they’re communicating with (eg, off-the-record chat)? Depending on what you’re trying to do, you will need different kinds of cryptosystems, and you will probably impose different requirements on what your product can actually do.

You should document which bits of data must be kept secret from, or authenticated to, which other parties, and how you intend to do that. You can then make sure that other features do not accidentally leak data that should be secret (eg, suggesting contacts to new users based on social networks of existing users may leak who people tend to contact). This also gives you a central place to check that ACLs are properly enforced; that any endpoint that serves data will properly check that the user is allowed to access every part of that data.

Use IVs

If you need to use block ciphers (such as AES) directly, never use ECB (Electronic Code Book) mode; it encrypts identical plaintext blocks to identical ciphertext and can leak information about common plaintext.

Instead, use CBC or CTR mode, which let you provide an additional input that will differentiate identical blocks.

Don’t reuse IVs

Following up from the previous point, in order to properly benefit from this protection, your IVs / nonces must be unique every time you encrypt.

A common beginner’s mistake is to either use a fixed IV, or generate an IV by hashing the key or the plaintext (or some other fixed value). This defeats (most of) the purpose of CBC; since two identical plaintexts use the same IV, they will encrypt to identical ciphertexts, leaking information again (using a fixed IV will scramble identical blocks in different plaintexts, but that alone is not enough).

Instead, you should generate a (from a cryptographically-secure RNG) a random IV for each message, and transmit it alongside the ciphertext.

Don’t expose error messages or other internal information

When dealing with crypto code, never show raw error messages to the user. Detailed error messages from crypto libraries should go in (access-controlled) logs. Your user-facing error messages should just say things like “Invalid input” and not give any details about why. If you expose actual error messages like “Invalid block size” or “Bad padding at byte X”, cryptanalysts may be able to slowly deduce bits of plaintexts or even keys, using techniques like padding oracle attacks.

This also applies to error messages resulting from decrypted data. If you parse a decrypted plaintext as XML, showing a detailed XML parsing error can help the attacker see exactly how many bytes he got right, and in some cases can even leak actual plaintext (from more-useful error messages that include the invalid line). If you expose these raw error messages, attackers may even be able to fully decrypt arbitrary non-XML ciphertexts by reading the XML parse errors.

Similarly, if you decrypt a ciphertext and validate the data using some business rules, detailed error messages from those rules can also leak information. Even if you only show these errors to authenticated, trusted users, you may have subtler risks, such as allowing users to decrypt parts of other users’ data, or attackers who manage to exfiltrate your error messages from those trusted users (eg, by posing as tech support).

Validate everything

Don’t be liberal with what you accept. When creating public APIs, it can be helpful to accept all requests, even if they don’t exactly match what you’re expecting. However, when dealing with cryptography or with sensitive information, this is exactly the attitude you don’t want.

Security-critical code should validate all attacker-controllable inputs as strictly as possible. Any invalid or unexpected data should be rejected immediately with no further processing, and with completely generic error messages. When possible, you should sign (with a MAC) all ciphertexts, so that you can reject any data that an attacker has modified before it has a chance to interact with anything else (and potentially expose weaknesses or side-channels in your decryption code).

Like detailed error messages, accepting invalid data can allow attackers to slowly learn parts of your encryption keys, using padding oracle attacks or timing vulnerabilities to see minute differences in how your system handles different inputs. Rejecting modified inputs immediately prevents these attacks.

Note: Although I have some understanding of the use of cryptographic primtiives, I am not a professional cryptographer.
Please take any cryptographic advice with a grain of salt (pun intended). If you are building highly sensitive systems, you should verify with your company’s cryptographer (you do have one, right?)

Chains of Trust – The Root of all Security

2015-10-31T00:00:00+00:00

The entire field of computer security is dedicated to verifying the source, confidentiality, or integrity of information or communication. This includes guaranteeing that the web page or other resource you’re seeing actually came from the entity you’re trying to reach (and has not been modified); guaranteeing that information you transmit will only be readable by that entity, or that the user connecting to a server is actually in possession of a token.

This raises a problem. Security may be all about trust and authentication, but how can you establish that trust in the first place? Without some existing indicator of identity, all the cryptography in the world cannot tell whether the data you’re receiving came from Bob, or from Mallory pretending to be Bob. If you start from a blank slate, any kind of proof you could ask Bob for can easily be duplicated by Mallory, leaving you none the wiser.

To solve this chicken-and-egg problem,

Bootstrapping - voice authentiation for OTR chat

The Web Authentication Arms Race – A Tale of Two Security Experts

2015-10-13T00:00:00+00:00

Web authentication systems have evolved over the past ten years to counter a growing variety of threats. This post will present a fictional arms race between a web application developer and an attacker, showing how different threats can be countered with the latest security technologies.

This entire conversation assumes that the user has already legitimately established some form of trust anchor (eg, a password or hardware token) with the defender before the attacker came onto the scene. Cryptography can only be used to transfer existing trust or secrecy across time or space; if the attacker impersonates the defender before the user establishes anything, it becomes impossible for the user to tell which party is legitimate. This also assumes that the site itself has no vulnerabilities (such as XSS or CSRF) that would allow attackers to run code or read data, or read certificates from the server.

Defender: Users will enter a username & password, and I will give them an authentication cookie for me to trust in the future.
Attacker: I will watch your network traffic and steal the passwords as they come down the wire.
Defender: I will change the <form> to submit over HTTPS, so you won’t see any readable passwords.
Attacker: I will run an active MITM attack as the user loads the login page, and insert Javascript that sends the password to my server in the background.
Defender: I will serve the login page itself over HTTPS too, so you won’t be able to read or change it.
Attacker: I will watch your network traffic and steal the resulting authentication cookies, so I can still impersonate users even without knowing the password.
Defender: I will serve the entire site over HTTPS (and mark the cookie as Secure), so you won’t be able to see any cookies.
Attacker: I will run an active MITM attack against your entire site and serve it over HTTP, letting me see all of your traffic (including passwords and cookies) again.
Defender: I will serve a Strict-Transport-Security header, telling the browser to always refuse to load my site over HTTP (assuming the user has already visited the site over a trusted connection to establish a trust anchor).
Attacker: I will find or compromise a shady certificate authority and get my own certificate for your domain name, letting me run my MITM attack and still serve HTTPS.
Defender: I will serve a Public-Key-Pins header, telling the browser to refuse to load my site with any certificate other than the one I specify.

At this point, there is no reasonable way for the attacker to run an MITM attack without first compromising the browser.

Attacker: I will make a fake login page and phish users for passwords.
Defender: I will add two-factor authentication, making your stolen passwords useless without the non-reusable second factor.
Attacker: I will change my phishing page to request a second factor as well, then immediately use it to log in once. (this will give the attacker a single login session with no way of logging in again, but that is often enough to cause harm)
Defender: I will replace my SMS or TOTP second factor with a private key on a tamper-resistant hardware device, rendering an MITM attack completely unable to use the stolen credential (the private key is used to sign a challenge from the server, and never leaves the device). This also prevents phishing attacks, since the browser will incorporate the site origin into the challenge signed by the private key, and will refuse to send a challenge signed for the defender’s server to any other origin. This is only possible because the browser actively cooperates, unlike purely web-based solutions like SQRL.

Private keys, such as U2F devices, are unphishable credentials; it is now completely impossible for anyone who does not have physical posession of the private key to authenticate. Note that this assumes that the hardware device is trusted; if the attacker can swap the device for a device with a known private key, all bets are off. Also note that you should still use a password in conjuction with the hardware device, to prevent an attacker from simply stealing the device (if the device itself requires a password to operate, that’s also fine).

Attacker: I will trick the user into installing a malicious browser extension or desktop application, then use it to read the authentication cookie from the browser’s cookie jar.
Defender: I will use channel-bound cookies, linking my authentication cookie to the private key used to generate the SSL connection. This way, the authentication cookie will only work in an HTTPS session backed by the same private key, preventing the attacker from using it on his computer.
Attacker: I will change my malicious code to exfiltrate the private key as well as the authentication cookie, allowing me to completely clone the SSL connection on my machine, and still use the cookie.
Defender: I will hope that the user’s browser signs its HTTPS connections with a hardware-based private key (hardware-backed token binding), preventing the attacker from cloning the SSL session without access to that private key (which never leaves the hardware device).
Attacker: I will change my malicious code to run a reverse proxy through the user’s browser, sending my arbitrary requests through the same token-bound SSL session as the user’s actual requests.
Defender: I will encourage users to use a platform & browser that does not allow processes or extensions to interact with security contexts for other origins. This way, the attacker’s malicious code will not be able to read my cookies or send requests to my site.

Assuming no application-level vulnerabilities (such as XSS or CSRF), and no vulnerabilities in the platform itself, such a platform would be completely secure against any kind of attack. Unfortunately, I am not aware of any such platform that also supports unphishable credentials. Chrome OS supports unphishable credentials, but offers no way to prevent extensions from sending HTTP requests to your origin. Most mobile browsers (on non-rooted devices) do not support extensions at all, but do not currently support unphishable credentials.

Theory vs. Practice

In theory, given these assumptions, this design is completely secure. In practice, however, the situation is rarely so simple. There are a number of issues which make these assumptions difficult to meet, or negate security in other ways.

Authenticating the user

This technique serves to completely prove that the user is in physical possession of a hardware token. However, it does not help assure you that the hardware token actually belongs to the user you think it does. If an attacker can convince you that his hardware token belongs to the user, he can still perform MITM attacks, since your server has no way of knowing that he isn’t actually the user.

By definition, you cannot use a hardware token to authenticate the user who is setting up a hardware token, so you’re back to relying on classical HTTPS and certificate pinning, leaving you vulnerable to malicious code running on the client. If you’re associating an online account with some existing entity, such as a bank account (as opposed to creating an entirely new account, such as an email account), you’ll need some way of proving that the user creating the account is actually associated with the existing entity, and not an attacker impersonating or MITMing the user. The best way to do this is for the user to verify themselves in person and associate the token using trusted, defender-controlled hardware.

Trusting hardware

It is impossible to defend against an attacker who can run arbitrary code on the user’s machine (since there is no way for the defender’s server to distinguish between requests sent from legitimate code on the user’s behalf, vs. hostile code from the attacker). The completely-secure scenario above postulated that the user’s platform does not allow arbitrary code in the first place, neatly side-stepping this problem.

Mobile platforms attempt to make this guarantee, preventing other applications from reading the browser’s cookie jar or private keys. However, completely enforcing this is harder; any security vulnerability (or actual nefarious code) in the browser, OS, or even platform hardware may negate this guarantee. For example, the ability to root or jailbreak an Android or iOS device (and thus bypass the protections around the browser) mean that they do not actually provide this guarantee.

An important step forward in this regard is trusted boot, as used by Windows and Chrome OS, in which the entire chain of execution from the BIOS to the boot loader to the OS to the hardware drivers is signed and verified on each boot. However, security vulnerabilities in the signed code can still lead the arbitrary code execution and compromise the machine.

More troublingly (assuming nation-state level attackers), even if the user does have completely-secure hardware, the attacker can replace it with attacker-controlled hardware that contains a reverse proxy and bypasses all of these protections. There is no simple way for the user to authenticate their own hardware device (other than keeping it in sight at all times); the attacker can open up the existing device and copy all data and private keys to the backdoored device to ensure that it looks and behaves identically.

(You could build a tamper-proof hardware-backed private key that will self-destruct if removed from the PCB, but you’ll need to hope that your tamper-proofing is better than the attacker’s tampering. And that the attacker can’t modify other parts of the hardware to insert a backdoor without removing anything.)

Forgot password

Finally, the Forgot Password feature is the complete antithesis of these security guarantees. The entire point of Forgot Password is to authenticate the user without any of these trusted credentials. Thus, this reintroduces the problem of initially associating the user, as mentioned in the first issue. In this situation, unlike account creation, you always need to prove who the user is, since you’re associating the user with an existing account, rather than simply creating a new account from scratch.

Forgot Password features are typically implementing by delegating authentication to some other (implicitly-trusted) service; usually email. When first creating the account, the user specifies an email address, and the user can then authenticate by proving that he can read an email sent to that address (usually alongside a low-grade second factor such as a security question). However, this opens up additional risks; any attacker that can compromise the user’s email account (and the second factor, which is usually easy) can compromise the account at any time. In addition, if the user legitimately uses the Forgot Password, there is no way to prevent malicious code already running on the client from MITMing that connection and compromising the account (eg, by adding the attacker’s private key alongside the user’s).

Handling margins in CSS

2015-08-20T00:00:00+00:00

Writing accurate, maintainable CSS for large websites requires careful planning and meticulous attention to detail. One detail that can often slip through the gaps is proper spacing between elements. If a view has a number of independent, optional components laid out in a vertical (or horizontal) stack, it can be tricky to ensure that each piece has correct spacing around it in all configurations.

I’ve found that

Code Snippets: Impossible Code

2015-06-30T00:00:00+00:00

This post is part of a series of blog posts called code snippets. These blog posts will explore successively more interesting ways to do simple tasks or abuse language features.

I recently set out to create snippets of code that have nothing inherently wrong with them, but can never appear in a valid program.

Impossible accessibility

The simplest example is a statement that uses internal types from two different assemblies, so that there is no project that it could legally appear in:

// In A.dll:
public class Base {
    internal static void Method() { }
}

// In B.dll:
public class Container {
    public class Derived : Base { }
}

Container.Derived.Method();

Note that the method itself is not impossible to call; Base.Method() is perfectly legal anywhere in the first assembly. However, the specific expression Container.Derived.Method() is impossible, since it uses types from both assemblies.

Adding an [InternalsVisibleTo] attribute to the first assembly, or compiling both assemblies with circular references (this is possible, if you try very hard), would make this statement legal. To prevent that, you can simply make both nested members protected.

void expressions

A more interesting approach is to create an expression of type void. This expression is completely fine, except that there is no way to use an expression of type void anywhere.

The only valid place that the type void can appear in C# is as a method return type; thus, the only way to make an expression of type void is a method call. Method calls are legal as expression statements, so that is perfectly fine.

However, there is one feature in the language that lets you call an expression with an arbitrary return type, in a syntax which is not legal as an expression statement: LINQ queries.

class Impossible {
    public static void Select(Func<int, int> a) { }
}

from x in Impossible select 1;

This would compile to Impossible.Select(x => 1), which is a perfectly fine expression statement, returning void. However, query expressions, unlike method call expressions, are not valid as expression statements, so this call is illegal.

Any attempt to use this query will result in either “CS0201: Only assignment, call, increment, decrement, and new object expressions can be used as a statement”, “CS0127 Since ‘Method()’ returns void, a return keyword must not be followed by an object expression”, or “CS0815: Cannot assign void to an implicitly-typed variable”. (except for a bug in the RC release of the Roslyn compiler)

Side note: I need to take the unusual approach of making a LINQ query against a static method on a type to prevent people from making it compile by adding a matching extension method that does not return void.

Impossible overload resolution

A subtler approach is to make two overloads of a generic method such that it is impossible to distinguish between them:

void Method<T, U>(T p) { }
void Method<T, U>(U p) { }

There is no way to call either of these methods with the same type for both generic type parameters (eg, Method<int, int>).

Ordinarily, C# has a number of ways to help disambiguate between generic overloads. You can cast ambiguous parameters to the exact argument type of the desired overload. You can explicitly specify type parameters. You can call the method through a specific qualifier (to disambiguate between static and instance methods, or between methods inherited from a base class). You can even pass named parameters if the two overloads have different parameters names.

However, these methods are identical in all of these regards, leaving no way to tell the compiler which one you want to call. (note that it is possible to call them at runtime using Reflection)

Generic constraint conflict

Another option is to create a method with conflicting generic type constraints, such that there is no possible type that will satisfy the constraints. Ordinarily, the compiler will not let you do that; a type parameter cannot be constrained to two different classes. However, you can bypass this restriction using more generics:

class Container<T, U> {
	public void Method<Z>() where Z : T, U {}
}

It is impossible to call Method on any instance parameterized with classes that do not inherit one another (eg, Container<Exception, Type>). Unlike the previous example, it is completely impossible to call this method on such an instantiation, even with reflection – there is no type that is valid value for Z.

Impossible inheritance

Any of these impossible methods can also be used to make a class that can never have any instances, by making an abstract method that cannot be implemented:

abstract class Base<T, U> {
    public abstract void Method(T x);
    public abstract void Method(U x);
}
abstract class Impossible : Base<int, int> { }

Any attempt to create a concrete non-abstract subclass of Impossible will result in “CS0462 The inherited members Base<T, U>.Method(T) and Base<T, U>.Method(U) have the same signature in type ‘Impossible’, so they cannot be overridden”.

abstract class Base<T, U> {
    public abstract void Method<Z>() where Z : T, U;
}
abstract class Impossible : Base<Type, Exception> { }

Here, implementing Impossible will give “CS0455 Type parameter ‘Z’ inherits conflicting constraints ‘Exception’ and ‘Type’”.

Interface method conflict

Since Java is less feature-rich than C#, there are far fewer examples of impossible code in Java. However, this feature sparsity can be used to create code which is only impossible in Java – inheriting conflicting interface methods:

interface A {
    int method();
}
interface B {
    long method();
}
final class C implements A, B { }

In Java, it is impossible to implement C. In C#, this would be possible with explicit interface implementation, which essentially implements each method with a private (and therefore unique) compiler-generated name, avoiding problems of identical overloads. However, since Java does not support this feature, you’d need to make two methods with the same signature but different return types, which is impossible.

Side note: I need to add final because otherwise, making C abstract would let it compile without implementing the method.

If you can think of other examples of impossible code, please comment!

Code Snippets: Fast Runtime Property Access with Reflection

2015-06-26T00:00:00+00:00

This post is part of a series of blog posts called code snippets. These blog posts will explore successively more interesting ways to do simple tasks or abuse language features.

Reflection is great for accessing all properties (or an arbitrary property named at runtime) of an arbitrary type. However, Reflection has performance costs which can be unacceptable in critical codepaths. Instead, you can add an upfront cost and create generic delegates to read or write such properties without any overhead at all (depending on what you’re doing with the values, you can even avoid boxing).

The typical goal for this kind of code is to get the value of a property with the specified name.

The naive (and slowest) way to do this is straight-up reflection:

string name
object o;

object value = o.GetType().GetProperty(name).GetValue(o);

This does all of the work of locating the property and dynamically invoking its getter on every call. You can improve this by caching each PropertyInfo:

var properties = new Dictionary<string, PropertyInfo>();

PropertyInfo property;
if (!properties.TryGetValue(name, out property))
	properties.Add(name, (property = o.GetType().GetProperty(o)));
value = property.GetValue(o);

(if multiple threads are involved, you’ll need either [ThreadStatic] or a ConcurrentDictionary)

However, this still repeats the work of dynamically invoking the getter method. You can eliminate this by creating a delegate that points to the get method. Creating the delegate still involves some costly reflection, but once it’s created, calling it will be as fast as calling any other delegate.

var getters = new Dictionary<string, Func<T, object>();

Func<T, object> getter;
if (!getters.TryGetValue(name, out getter)) {
	getter = (Func<T, object>)Delegate.CreateDelegate(
		typeof(Func<T, object>),
		typeof(T).GetProperty(name).GetMethod
	);
	getters.Add(name, getter);
}

T o;
value = getter(o);

This uses delegate return type covariance to create a Func<..., object> from a getter method that returns any more-derived type. This feature actually predates generic covariance; it will work even on .Net 2.0.

This approach has some caveats. This can only work if you know the type of the objects you’re operating on at compile-time (eg, in a generic method to bind or serialize parameters). If you’re operating on objects at compile time, this cannot work, since such a delegate wouldn’t be type-safe.

This also won’t work for properties that return value types. Variance works because the runtime representation of the types are completely identical, so that the JITted code doesn’t know or care that the actual type parameter is different. Value types require different codegen than reference types, so this cannot begin to work.

To solve these problems, you can do actual runtime code generation, using the ever-handy Expression<T>. Specifically, you need to create an expression tree that casts an object parameter to T (if you don’t know the actual type at compile time), then boxes the result of the property into an object (if it’s a value type).

var param = Expression.Parameter(typeof(T));
getter = Expression.Lambda<Func<T, object>>(
	Expression.Convert(
		Expression.Property(param, name),
		typeof(object)
	),
	param
).Compile();

The Expression.Convert() node will automatically generate a boxing conversion if T is a value type, making this work. This whole block is only necessary if T is a value type; if it’s a reference type, you can skip the entire runtime codegen phase with the previous example.

Code Snippets: And in the darkness bind() them

2015-06-18T00:00:00+00:00

This post is part of a series of blog posts called code snippets. These blog posts will explore successively more interesting ways to do simple tasks or abuse language features.

Javascript’s bind() method (new to ES5) is very useful for passing callbacks or event handlers that use an existing this.

It also has some more confusing uses, especially when combined with call() or apply().

If you have an array of functions, and you want to execute every function in the array, you can write

myFuncs.forEach(function(f) { f(); });

However, all you’re really doing is calling .call() on every function in the array. Therefore, you would instead want to write

myFuncs.forEach(Function.prototype.call);

However, this won’t work, because call() calls the function in its this parameter (ie, myFunc.call()), not its first argument. This code would end up running

Function.prototype.call.call(null, myFuncs[0], 0, myFuncs);

This will throw a TypeError, since call() can only be called on a function.

What you actually want to run is

Function.prototype.call.call(Function.prototype.call, myFuncs[0], ...);

In other words, you want to call call() and pass your function as its this (before any other parameters). To make forEach() do that, you need to bind call() to itself:

myFuncs.forEach(Function.prototype.call.bind(Function.prototype.call));

To make this code shorter (and even more confusing), note that since Function is itself an instance of Function, you can skip the prototype and reference call directly:

myFuncs.forEach(Function.call.bind(Function.call));

You can make it even shorter by referencing call from a shorter function:

myFuncs.forEach(eval.call.bind(eval.call));

(other short function names, such as Date or isNaN, would also work)

Side note: Please, don’t do that.

This technique effectively “uncurries” the this parameter from a function. It takes call(), which accepts the function to call as this, and produces a function that accepts the function to call as its first normal parameter (and ignores this). You can use the same technique for other prototype functions. For example, you can turn Array.push into a standalone function that mutates its first parameter:

var pushInto = Function.call.bind(Array.prototype.push);

var myArray = [1, 2];
pushInto(myArray, 3);
console.log(myArray); // 1,2,3

Or you can take an array of strings and turn them to uppercase:

["a", "b"].map(Function.call.bind(String.prototype.toUpperCase));

You can also do even more confusing things by bind()ing bind itself. For example, you can simplify the above examples by wrapping the bound call in its own function:

var uncurryThis = Function.bind.bind(Function.call);
var copyArray = uncurryThis(Array.prototype.slice);

var array = [1, 2, 3];
var copy = copyArray(array);

This works by binding bind() to have call as its this parameter, so that the resulting function becomes call.bind(). I’ve seen this trick in actual production code, in the most popular Javascript promise library (it was eventually replaced with an explicit version for performance reasons).

You can take an array of objects, and turn it into an array of functions bound to those objects:

var elements = [document.head, document.body];
var cloners = elements.map(Function.bind.bind(Element.prototype.cloneNode));
var clone0 = cloners[0]();

This takes an array of elements and creates an array of functions that will clone those elements on each call. It works by binding bind() to bind cloneNode(); it ends up calling cloneNode.bind(array[x]).

Side note: With great power comes great responsibility. Just because you can write code like this doesn’t mean you should.

Code Snippets: Variadic Generics in C#

2015-06-16T00:00:00+00:00

This post is part of a series of blog posts called code snippets. These blog posts will explore successively more interesting ways to do simple tasks or abuse language features.

C++ introduced Variadic Templates – template classes or functions that can take an arbitrary number of template parameters (like varargs/paramarray function parameters). This feature has a number of uses. It’s the simplest way arbitrary tuple or function types. It’s also useful when making a function that can take an arbitrary number of objects or delegates.

C# does not support this feature. For variadic types, there is no direct workaround; this is why the BCL has 16 overloads of the generic Func and Action delegates, and 8 overloads of Tuple, to cover all common uses.

However, when designing something that needs to take an arbitrary number of objects of different generic parameters, there are workarounds.

This simplest workaround is to use repeated method calls. You can make a generic function that returns this, and call it separately for each parameter with inferred type parameters. For example:

class Container1 {
    public Container1 WithStuff<T>(Func<T> func) {
        // Do interesting things...
        return this;
    }
}

new Container1()
    .WithStuff(() => new List<byte>())
    .WithStuff(() => new MemoryStream())
    .WithStuff(() => new CredentialCache());

The disadvantage of this approach is that the repeated method calls don’t look nice, especially if you’re just trying to accept a number of parameters for a single thing.

A more interesting approach is to abuse collection initializers. Collection initializers let you write a list of expressions (or argument lists) in braces (just like an array initializer), and compile them into a series of Add() calls. The nice part of this feature is that each item compiles to its own Add() call (with its own generic parameters and type inference), giving you a way to hide the repeated calls from the syntax.

The previous example can thus be adapted to look like this:

class Container2 : System.Collections.IEnumerable {
    public void Add<T>(Func<T> func) {
        // Do interesting things...
    }

    System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator() {
        throw new NotSupportedException();
    }
}

new Container2() {
    () => new List<byte>(),
    () => new MemoryStream(),
    () => new CredentialCache()
};

The class must implement the non-generic IEnumerable interface in order to allow collection initializers (since the feature is meant, as its name implies, to work with collections).

Unlike normal method calls, collection initializers don’t expose any way to explicitly specify generic type parameters; they can only be called using type inference. For methods that accept instances of T (or lambdas that return such instances), this is not a problem; the compiler can infer the type parameter from the compile-time type of the expression used. If your method only uses the generic type parameter as the parameter type of a lambda, you must instead explicitly specify the parameter type in the lambda (eg, (string x) => blah), telling the compiler what T is.

Note that this only works at construction time (syntactically, this is an optional part of the new keyword). If you’re trying to make a variadic method on an existing instance (or a static method), you can make a new class to serve as the method’s parameter, and put the collection initializer in that class. For example:

class ValidationManager {
    readonly List<Tuple<object, string>> errors;

    public void ValidateItems(ItemValidators parameter) {
        errors.AddRange(parameter.Errors);
    }
}

class ItemValidators : System.Collections.IEnumerable {
    readonly List<Tuple<object, string>> errors;
    public IReadOnlyCollection<Tuple<object, string>> Errors => errors.AsReadOnly();
    public void Add<T>(string error, T instance, Func<T, bool> rule) {
        if (!rule(instance))
            errors.Add(Tuple.Create((object)instance, error);
    }

    System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator() {
        throw new NotSupportedException();
    }
}

var validation = new ValidationManager();

validation.ValidateItems(new ItemValidators {
    { "Stream cannot be empty", new MemoryStream(), s => s.Length == 0 },
    { "Email address must have a display name", new MailAddress(), m => string.IsNullOrWhiteSpace(m.DisplayName) }
});

As you may have noticed from this example, these workarounds don’t help you store variadically-typed items. What you do with the parameters you receive depends on what you’re ultimately trying to accomplish. If you’re dealing with reflection (but want strongly typed lambdas), you can simply store typeof(T). If you just need instances (like the above example), you can cast the items to object. However, the most powerful way to deal with this is to collect non-generic delegates that close over the generic parameter (creating a generic closure class). As long as you’re able to do useful things from a non-generic entry-point (eg, cast an object to T), you can retain the generic parameter from within the delegate. For example:

class ItemReporter : System.Collections.IEnumerable {
    readonly Dictionary<Type, Func<object, string>> reporters
        = new Dictionary<Type, Func<object, string>>();


    public void Add<T>(Func<T, string> reporter) {
        reporters.Add(typeof(T), obj => typeof(T) + "\t" + reporter((T)obj));
    }

    public string Report(params object[] items) {
        return string.Join(Environment.NewLine,
            items.Select(obj => reporters[obj.GetType()](obj))
        );
    }


    System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator() {
        throw new NotSupportedException();
    }
}

var reporter = new ItemReporter {
    (FileStream fs) => fs.Name + "; " + fs.Length + " bytes",
    (DateTime d) => d.ToString("U"),
    (DirectoryInfo d) => d.Name + ": " + d.EnumerateFiles().Count() + " files"
};
Console.WriteLine(reporter.Report(
    DateTime.Now,
    DateTime.UtcNow, 
    new DirectoryInfo(Environment.CurrentDirectory)
));

As mentioned above, you must explicitly specify the lambda parameters to allow the compiler to infer T for each generated Add() call.

This demonstrates one more limitation of this workaround: It is impossible to store type information. If you implement this class in C++, you’d be able to enforce – at compile time – that the arguments passed to Report() match the types of the reporters passed to the constructor (since those types would be encoded in the template parameters of the instance’s compile-time type). In C#, however, the only way to implement Report() is to accept an array of object, which obviously has no type safety.

Note that this approach does not work for base classes or interfaces; if the reporter doesn’t have a delegate for an object’s exact runtime type, the dictionary lookup will fail. It is impossible to allow that with a dictionary (since assignment compatibility is not an equivalence relation). If you want to support that, you could either loop through every reporter until you find one with a compatible type (which would be O(n) in the number of reporters, but would be simpler code), or recursively loop through the base classes and interfaces of obj.GetType() until you find a type in the dictionary (which would be O(n) in the depth of the type hierarchy).

This technique is particularly useful for visitor patterns (example usage and implementation); more details coming soon in a separate blog post.

Code Snippets: Conditional Casting

2015-06-12T00:00:00+00:00

This post is the beginning of a new series of blog posts called code snippets. These blog posts will explore successively more interesting ways to do simple tasks or abuse language features.

When writing C#, you will occasionally need to check whether an object is an instance of a certain type (eg, Button), and use part of that type if it is.

This is most commonly written as

object x = ...;
if (x is Button) {
	var b = (Button) x;
	// Use b...
}

This approach is simple and readable. However, it has an unnecessary performance hit. .Net casts are not free; every cast instruction must check that the object being casted is in fact convertible to that type. Casting the same object twice will duplicate that check for no good reason, and should be avoided.

The recommended pattern for this is to pull the as check outside the if:

var b = x as Button;
if (b != null) {
    // Use b....
}

However, this code has the disadvantage of leaking b into the containing scope, allowing you to accidentally use it outside the if block when it might be null.

A clever (but less readable) alternative is

for (var b = x as Button; b != null; b = null) {
	// Use b...
}

(Credit to Jon Skeet)

This works by abusing the elements of the for loop, and, in particular, taking advantage of the fact that the variable declared in the for statement is within its own scope. The first clause simply declares the b variable, and initializes it to be either null or the correct type. After running that, the compiler will immediately run the second clause (the termination condition). Thus, if b is null (if the cast failed), it will exit the “loop” immediately. Finally, after the body of the code block, it will run the third clause, setting b back to null and ensuring that the loop will not run again.

For simple uses, C# 6 introduces the ideal way to do this with the conditional access operator:

(x as Button)?.Click();
(null as StreamWriter)?.WriteLine();

If the cast fails, it will evaluate to null, and the ?. operator won’t do anything. However, this only helps if you’re doing exactly one method call on the casted expression, and does not work for assignments or event handler registration.

Side note: This should generally be avoided; instead, consider using a virtual method on the base class, or a visitor pattern.

Concurrency, part 6: Easier asynchrony in C# with await

2015-06-11T00:00:00+00:00

Last time, I described more advanced patterns for complicated workflows involving asynchronous operations. These patterns can be annoying to write. However, modern compilers can bear the brunt of this complexity, allowing you to write code as if the operations were synchronous, then letting the compiler transform your code into mes promise chains.

C# 5 introduces this with its flagship new async / await keywords. However, the wealth of possibilities opened by this feature has left many developers confused about when to make async methods and when not to.

Non-blocking IO

The async keyword does not create asynchrony; instead, it allows you to call existing asynchronous methods easily. If your code isn’t calling any existing *Async() methods (which return Task or Task<T>), async won’t do any you good. However, if you have the option of calling synchronous or asynchronous versions of an operation (eg, StreamWriter.WriteLine() vs. StreamWriter.WriteLineAsync()), you can use await to switch your code to use asynchronous operations, gaining all of the scalability benefits of non-blocking IO without any of the complexity.

Where do asynchronous operations come from?

The point of an asynchronous operation is to wait for something to happen without wasting any threads. For example, if you’re waiting for a web service to send you a reply, you can use an asynchronous HTTP client to wait for that reply and release the thread you were running on to do other things while you wait. To make this happen, the operating system provides an asynchronous networking API that will wake up and call your asynchronous continuation when a response arrives from the network card.

In other words, an operation can be made asynchronous if it simply waits for something to happen elsewhere, rather than actively computing something. Operations like reading or writing from a hard disk or network, or waiting 5 seconds, are all inherently asynchronous; they do not involve any direct work beyond sending a request to the hard disk or network card, or setting a timer interrupt.

By contrast, operations like adding a million numbers, or encrypting data, are not asynchronous. They may take a long time, but all of that time is spent actively computing something with the CPU, so you can’t release your thread and wait for something to happen on its own. (if you have a separate chip to perform cryptographic operations, this is not true)

For more details about this distinction, see part 1 of this series.

Therefore, you cannot use C#’s async keyword to create a new asynchronous operation; you can only create an asynchronous operation if you have some external event to wait for.

So what are async methods good for?

async methods are good for composing existing asynchronous operations – calling multiple operations in sequence and using the results. All of the complex promise code described in my previous post can be implemented much more easily using await.

For example, if you wanted to find out which of two places was warmer, you could write the following asynchronous code by hand:

var client = new HttpClient();

var placeTasks = new[] {
	client.GetAsync("https://api.weather.example.com/Locations/" + Uri.EscapeUriString("New York, NY")),
	client.GetAsync("https://api.weather.example.com/Locations/" + Uri.EscapeUriString("Seattle, WA"))
};
	
	// Wait for both tasks to finish - for the response headers to start arriving
Task.WhenAll(placeTasks).ContinueWith(task =>
	task.Result.Select(response => response.Content.ReadAsStreamAsync())
	// Wait for the response bodies
).ContinueWith(tasks => {
	var xmlBodies = tasks.Result.Select(t => XDocument.Parse(t.Result));
	var hottest = xmlBodies
		.Select(x => (decimal)x.Descendants("Current").Single().Element("Temperature"))
		.Max();
});

Using await makes this much simpler:

// Wait for both tasks to finish - for the responses to start arriving
var responses = await Task.WhenAll(placeTasks);

// Wait for the response bodies
var bodies = await Task.WhenAll(responses.Select(response => response.Content.ReadAsStreamAsync()));

var xmlBodies = bodies.Select(t => XDocument.Parse(t.Result));
var hottest = xmlBodies
	.Select(x => (decimal)x.Descendants("Current").Single().Element("Temperature"))
	.Max();

Behind the scenes, the compiler will transform async methods into a series of promise callbacks, storing your local variables in a mutable closure class. (It’s actually more complicated than that; Jon Skeet explained exactly how it works in his Eduasync blog posts)

This feature is most helpful for more complicated code, especially when consuming operations in sequence. For example, the complicated reduce() loop (from my previous blog post) to asynchronously process an array of items in sequence becomes much simpler:

foreach (var item in items) {
	await SomeFunctionAsync(item);
}

To gather the results, you can simply add them to a list:

var results = new List<T>();
foreach (var item in items) {
	results.Add(await SomeFunctionAsync(item));
}

Pitfalls

The most common mistake with async methods is async void. An async method is required to return one of void, Task, or Task<T>. If your function doesn’t return a value, it can be tempting to make it return async void. The problem with this option is that it makes it impossible for the caller to find out when the asynchronous portion of the method is finished, or to handle any errors that are thrown asynchronously. As I explained in part 2, in order to wait for an asynchronous operation to finish, it must return a promise.

The only correct place to use async void is event handlers. Since the code that raised the event doesn’t care when you finish handling it, there is nothing wrong with hiding the promise from it. (In fact, event handlers must be async void, since event delegates are declared as returning void.)

Concurrency, part 5: Advanced Promise Usage

2015-06-10T00:00:00+00:00

Last time, I listed standard & third-party promise libraries in popular languages, comparing how each one implements the promise paradigm. In this post, I’ll describe some advanced patterns for promise usage.

Error handling

One of the more useful features of promise-based asynchrony is automatic propagation of errors. However, just like traditional exception handling, this feature is only useful if errors are correctly propagated up the call stack until they reach a method that knows how to handle them.

Promise-based error handling adds an additional concern in that errors are only passed along explicit promise chains. If you write a function that creates a promise chain and ignores it (without returning the resulting promise to its caller), errors in that chain will typically be silently ignored; this can hide serious bugs in your code.

These concerns translate into a simple set of guidelines. There are two kinds of methods that may involve errors: Library methods, which will throw errors, but do not know how to handle them, and application methods, which typically will not throw errors of their own, but do know how to handle errors from library methods that they call. Given this distinction, the guidelines are as follows:

Do not swallow errors. If you call a method that returns a promise, you should either handle errors from that promise directly (for application methods), or return the promise to your caller so that it can handle errors elsewhere.
This is not a hard-and-fast rule. If a library method invokes an operation and knows that no-one can possibly care about any failures (for example, if it tries two equivalent service endpoints, and one fails and the other succeeds), it may swallow the errors. If possible, it should log the errors to help application developers troubleshoot unexpected scenarios.
If a library method creates multiple promise chains, it can use Promise.all() to observe errors from any branch in the chains (consult the documentation for your promise library to find out how this behaves when multiple errors occur; many libraries have an alternative version of this method that will wait for every error).
If you don’t want to wait the method’s returned promise to wait or all of the promise chains to finish, you should probably refactor the method into multiple methods; each function should generally only serve one purpose. The caller can then call each operation separately and wait for its result or handle errors as appropriate.
Any reusable asynchronous function should (almost) always return a promise that consumes all of the function’s asynchrony, allowing callers to wait for the entire operation to finish and to observe any errors that occurred.
In C#, this means that you should never write reusable aync void methods (Async Sub in VB); instead, always use async Task, so that callers can wait for & observe the result.
Entry-point methods in applications should always handle errors from their promise chains & async method calls, displaying an error to the user and/or sending an error report to the developer.
For Javascript, the Q library has a useful done() helper method, which can be called at the tip of a promise chain to throw any errors and trigger the environment’s default unhandled error notifications (window.onerror in a browser, or the error event in process for Node.js). This is a convenient way to handle errors from promise chains in your application methods, especially if you already have a generic error handler for this event.
If a library method wants to add more context to an error, it can add an error callback that wraps the exception in a new exception with additional detail (making sure to preserve the inner exception details as well), then rethrows the new exception. This will reject the rest of the promise chain with the new error, just like throw new MySpecificException("...", ex) would inside a traditional catch block.

In summary, never leave a promise chain “dangling”. Instead, either return the resulting promise to your caller, or handle errors yourself at the end of the chain.

Calling asynchronous methods in loops

One common challenge when working with asynchronous operations is running an asynchronous operation in a loop over a collection of source items. Here too, promises can help simplify your code. The technique for doing this depends whether you want to run in sequence or in parallel.

Parallel Operations

If your operations are completely independent, and can run in parallel, you can simply kick off all of them at once, then wait for all of the promises to complete. In Javascript, this looks like this:

var items = [...];
var allDone = Promise.all(items.map(function(item) {
    return someFunctionAsync(item)
		.then(...);
}));

This code creates an array of promises from the array of items, then waits for all of them to finish. The resulting allDone promise will resolve to an array of the results of each of the original items. You can build a promise chain inside the map() callback to perform a whole sequence of operations on each item and have each sequence run in parallel. If you need to wait for all of the items to complete before continuing with further parallel operations, you can build a second set of parallel promise chains from the result of the first promise:

allDone.then(function(results) {
	var average = results.reduce(function(a, b) { return a + b; }) / results.length;
	return Promise.all(results.map(function(item) {
		return otherFunctionAsync(item, average)
			.then(...);
	});
});

This second set of promise chains will also run in parallel, but will only start after all of the first chains finish.

Sequential operations

Running a collection of asynchronous operations in sequence is more complicated. If you don’t want each operation to start until the previous operation finishes, you must build a chain of promises, like this:

var items = [...];
items.reduce(function(promise, item) {
    return promise.then(function() {
        return someFunctionAsync(item);
    });
}, Promise.resolve());

This code uses the Array.reduce() method to build up a chain of promises, executing each function after the previous one finishes. The first parameter to the callback is the promise returned from the previous iteration (for the first item, it’ll be the Promise.resolve() passed as the second parameter to reduce()). The reduce() callback adds a handler to that promise to execute the async operation on the current item, and returns the resulting promise, so that the next iteration will wait for it in turn. The effect looks like this:

Promise.resolve()
    .then(function() { return someFunctionAsync(items[0]); })
    .then(function() { return someFunctionAsync(items[1]); })
    .then(function() { return someFunctionAsync(items[2]); })
    ...

The value of the promise passed into the reduce() callback is the result of the asynchronous function from the previous iteration. Therefore, you can easily combine the current item with the result of the previous item.

If you need an array of the results of all of the operations, you need to build the array from the promise results (starting with an empty array in the initial promise):

var items = [...];
items.reduce(function(promise, item) {
    return promise.then(function(results) {
        return someFunctionAsync(item)
            .then(results.push.bind(results))
            .then(function() { return results; });
    });
}, Promise.resolve([]));

Passing state along a promise chain

When writing more-complex asynchronous workflows, you may need to pass state along a promise chain, from an intermediary promise result to a later promise callback. For example, you might need to asynchronously fetch a post, fetch its author, then render both objects.

Unfortunately, there is no good way to do this. The best option is to build an array of all of the objects you need, using Promise.all() to wait for new promises while keeping existing values:

loadPost(id)
    .then(function(post) {
        return Promise.all([post, loadAuthor(post.authorId)]);
    })
    .then(function(results) {
        var post = results[0];
        var author = results[1];
        // Do something with both values.
    })

Calling Promise.all() here will return a promise of the post and the author, which will be resolved once the author loads. This call is necessary because returning a simple array from a then() callback will not wait for the promises in the array. Without it, you would still have a promise of the author in the second callback.

In Javascript, the Q promise library has a spread() method to simplify this pattern. This method is like then(), but will flatten the array into individual parameters. It will even call all() for you, so you can return an array of promises and it will wait for all of them to be resolved first. It would simplify the above code:

loadPost(id)
    .then(function(post) {
        return [post, loadAuthor(post.authorId)];
    })
    .spread(function(post, author) {
        // Do something with both values.
    })

If you need to load multiple items in separate steps, you can keep building larger and larger arrays of all of the items you need to load (obviously, you should try to load them in parallel – by returning multiple promises in a single array – where possible).

Caching asynchronous operations

Another common task when writing asynchronous code is to cache the result of an asynchronous operation. As long as whatever you’re doing is reasonably idempotent (eg, loading data that rarely changes, or executing a fixed version of an external script file), you will generally want to load it just once, and have future calls reuse the first call. This technique is called memoization.

When doing this, you must be careful to avoid race conditions. If you only cache each call after the result arrives, you can still end up making multiple calls if the second call is made before the first one responds. Instead, you should cache the promise of the result immediately, so you can return that promise directly for the next call, whether it has loaded or not. However, if the call returns an error, you will presumably want to clear it from the cache so that the next call can try again (unless you know it’s a permanent error).

In Javascript, you might write the following code to memoize any single-argument asynchronous function:

function memoizeAsync(fn) {
	var map = Object.create();
	return function(arg) {
		if (map[arg]) return map[arg];
		return map[arg] = fn(arg)
			.thenCatch(function(error) {
				if (!isPermanent(error))	// Unless we know that it isn't worth retrying,
					delete map[arg];		// Remove this argument to try again next time.
				throw error;				// Rethrow the error so the result still fails.
			});
	};
}

In multi-threaded environments (ie, not Javascript), you must also beware of thread-safety. You can do this lock-free fashion by using a compare-and-swap loop to insert the promise into an immutable map. Make sure to only actually send the request after the compare-and-swap loop completes successfully, at which point you know that only your thread has added the value. (Depending on how your request call works, you may need to add a promise resolver to the map, then resolve it to the request result after sending it.)

Next time: Using await for easier asynchrony in C#

Challenges in Silon

2015-05-16T00:00:00+00:00

While writing Silon, I had to overcome number of challenges. You can see the full details in the comments & commit history of this repo, and in some cases in bugs I filed in the LESS compiler. Here are some of the more interesting ones.

##Nested XOR As described above, my operation mixins can take either simple selectors or further logical operations as their operand parameters. For the most part, this is simple. For AND operations, I can simply concatenate the parts together using sibling combinators (~). For OR operations, I can simply select each operand separately, and the LESS compiler will automatically expand every combination of alternates via selector nesting.

However, XOR is much harder. To implement an XOR selector, I need to invert each operand (a XOR b === (!a AND b) OR (a AND !b)).

##LESS Scoping

##Selector ordering

##Duplicate subparts

##Intermediate gates For this use, the most frustrating issue with CSS is the complete inability to reuse state as a selector. This is not a common problem.

##Browser bugs Chrome (only) will choke horribly if a CSS selector has more than a couple of hundred comma-separated parts.

##Checkbox styling To

##Drawing logic gates

Visual Studio Code under the hood

2015-05-03T00:00:00+00:00

Last week, Microsoft shocked the development community with Visual Studio Code, a new lightweight version of Visual Studio with full support for Mac & Linux. However, while Visual Studio Code may look very similar to the original Visual Studio, it’s actaully a complete rewrite, with almost nothing in common. This blog post will explore VS Code’s architecture in greater depth.

Like many recent products coming from DevDiv (Microsoft’s Developer Division), Visual Studio code is built almost entirely on open-source software (VS Code itself is not open source, but it soon will be).

Typename Comments, part 2: The method behind the madness

2015-04-02T00:00:00+00:00

Last time, I introduced a new syntax for code comments, typename comments. April Fools! However, although I do not intend for anyone to actually write code like this, everything I wrote in that blog post is true; this syntax does work.

The reason this technique works is that the letter classes are both contained in and derived from the outer Scope class. Because they’re contained in the Scope class, they are referenced as static members of that class – Scope.A. However, like any other static member, they can also be accessed through a qualifier of any class that derives from Scope. Since they themselves derive from Scope, they can be chained, writing Scope.A.B.C, etc. This is how we construct typenames that can be arbitrarily long

Nesting all of the actual classes in the project within the Scope class lets them be referenced in the same fashion. This is why you can write Scope.A.B.C.SomeRealClass. Since the actual classes don’t also inherit Scope, you cannot chain them further, but there is no need for that.

Finally, again since all of the actual classes are nested within Scope, you don’t need to qualify the typenames with Scope. – like any other static members, they can be accessed without a qualifier anywhere inside the declaring class.

This is why you can only use typename comments for types declared within your project (so they can be nested within Scope). Any other type cannot be referenced at the end of the chain, since it isn’t nested within the class at the tip.

Like other static members, the qualifier used to reference the member is purely syntactic sugar. Whether you write Scope.MyClass or Scope.S.o.m.e.L.o.n.g.C.o.m.m.e.n.t.MyClass, you’re actually just referencing Scope.MyClass (albeit in a convoluted way), so the comment is not visible in compiled code.

The syntax highlighting is actually a side-effect of an unrelated feature new to Roslyn. Roslyn will detect unnecessary qualifications – extra namespaces, using type names instead of keywords, or incorrect or unnecessary qualifiers for static members – and render them in a faded color, and also offer a quick fix to remove them. This entire technique is about adding unnecessary qualifiers to typenames, so Roslyn fades all of the qualifiers to tell you to remove them.

Introducing Typename Comments – A New Kind of Comment

2015-04-01T00:00:00+00:00

C# has two well-known styles of comments: C-style comments (/* ... */) and C++-style comments (// ...). This blog post introduces a third kind of comment: typename comments. With a bit of supporting code, you can write comments as part of type names – in casts, variable declarations, generic parameters, and anywhere else a typename can occur.

Typename comments are type expressions that precede any type that you define, like this:

partial class Scope {
	interface IPlayer {
		void PlayAgainst(M.u.s.t.B.e.A.n.N.P.C.IPlayer other);
		List<IPlayer> FindDefeatedOpponents();
	}

	class Player : IPlayer {
		W.i.l.l.B.e.R.e.s.e.t.A.t.E.a.c.h.L.e.v.e.l.Player opponent = null;
		readonly List<A.d.d.e.d.A.f.t.e.r.L.e.v.e.l.E.n.d.Player> pastOpponents = new List<Player>();
		bool defeated;

		public List<IPlayer> FindDefeatedOpponents() {
			return pastOpponents
				.Where(p => p.defeated)
				.ToList<C.l.a.s.s.e.s.C.a.n.n.o.t.B.e.C.o.v.a.r.i.a.n.t.IPlayer>();
		}

		public void PlayAgainst(IPlayer other) {
			opponent = (T.h.i.s.I.s.T.h.e.O.n.l.y.D.e.r.i.v.e.d.C.l.a.s.s.Player)other;
		}
	} 
}

You may have noticed the partial Scope class that contains all of this code. Unlike other comments, typename comments need this bit of supporting code. You can only use typename comments if both the type being referenced and the code containing the comment are wrapped in the same Scope class. In addition, you also need to add the following supporting code once in your project:

partial class Scope {
	class A : Scope { } class B : Scope { } class C : Scope { } class D : Scope { }
	class E : Scope { } class F : Scope { } class G : Scope { } class H : Scope { }
	class I : Scope { } class J : Scope { } class K : Scope { } class L : Scope { }
	class M : Scope { } class N : Scope { } class O : Scope { } class P : Scope { }
	class Q : Scope { } class R : Scope { } class S : Scope { } class T : Scope { }
	class U : Scope { } class V : Scope { } class W : Scope { } class X : Scope { }
	class Y : Scope { } class Z : Scope { }

	class a : Scope { } class b : Scope { } class c : Scope { } class d : Scope { }
	class e : Scope { } class f : Scope { } class g : Scope { } class h : Scope { }
	class i : Scope { } class j : Scope { } class k : Scope { } class l : Scope { }
	class m : Scope { } class n : Scope { } class o : Scope { } class p : Scope { }
	class q : Scope { } class r : Scope { } class s : Scope { } class t : Scope { }
	class u : Scope { } class v : Scope { } class w : Scope { } class x : Scope { }
	class y : Scope { } class z : Scope { }
}

Although these supporting classes are compiled into your code, the comments themselves, like any other comment, will be stripped by the compiler.

The best part of typename comments is their broad support spectrum: This syntax feature has been supported since the very first versions of both C# and VB.Net, over 13 years ago. However, the Visual Studio IDE only added syntax highlighting for typename comments with Roslyn in VS2015. Although it took them long enough to address this critical issue, you can now enjoy typename comments in their full glory:

Next time: The method behind the madness

Count() Considered Occasionally Harmful

2015-01-12T00:00:00+00:00

How many times have you seen code like this?

if (someSequence.Count() > 2) { }

LINQ’s [.Count() method](https://msdn.microsoft.com/en-us/library/vstudio/bb338038() has an important performance subtlety which can make this code unnecessarily slow. The root of the problem is that the IEnumerable<T> interface, which all LINQ methods operate on, doesn’t actually have a count (if it did, there would be no need for the Count() extension method in the first place). Therefore, the Count() method is actually O(n), looping through the entire collection to count every item.

Fortunately, the implementation of Count() is a bit smarter than this. As you can see in the source, Count() will check whether the source collection implements ICollection<T> or ICollection, and, if it does, will directly return the Count property without iterating at all (thus becoming O(1)). Unfortunately, it does not check for IReadOnlyCollection<T>; if you call Count() on a collection that implements IReadOnlyCollection<T> but not ICollection<T>, it will still be O(n). There are no such types in the BCL, but this has been enough of an issue for Roslyn that they wrote an analyzer to prevent it.

In fact, it can be even worse than that. It is perfectly possible to make an infinite IEnumerable<T> – a sequence that can be iterated as far as you want, without running out of elements. For example:

IEnumerable<int> RandomStream(int max) {
	var rand = new Random();
	while (true) yield return rand.Next(max);
} 

This sequence will work fine with most LINQ methods, because LINQ will only iterate through the sequence as far as it needs to. However, if you call any method that consumes the entire collection (such as Count(), Last(), OrderBy(), ToList(), and some others), it will either hang forever, or eventually throw an OverflowException or OutOfMemoryException, depending on exactly what the method does (Last(), for example, will hang without throwing anything).

The lesson from this mess is that Count() should be called with caution. If you actually need to know exactly how many items are in a sequence, there is no alternative; you must simply beware that if the source doesn’t implement ICollection<T>, it will be an O(n) call. Obviously, if the sequence is in fact infinite, this will not – and cannot – work.

If, on the other hand, you’re simply checking whether a sequence has at least N elements, like the code at the beginning of this blog post, there is no need to iterate over the entire sequence, and no reason for the code to fail on infinite sequences. All you need to do is iterate through the sequence, then stop immediately once you find N elements. You can do that in one line of LINQ:

if (someSequence.Skip(2).Any()) { }

However, this code is still not perfect. If the sequence does in fact implement ICollection<T>, this will be unnecessarily slow; instead of simply checking the Count property, it needs to instantiate two enumerators (one for each LINQ method), and iterate through the first N items in the sequence. In this scenario, checking Count() is in fact faster.

To get the best of both worlds, you need to write your own LINQ method:

public static bool HasMoreThan<T>(this IEnumerable<T> sequence, int count) {
	if (count < 0) return true;
	int? staticCount = (sequence as ICollection<T>)?.Count 
					?? (sequence as ICollection)?.Count
					?? (sequence as IReadOnlyCollection<T>)?.Count;
	if (staticCount != null)
		return staticCount > count;
	using (var enumerator = sequence.GetEnumerator())
		for (int i = 0; i < count + 1; i++)
			if (!enumerator.MoveNext())
				return false;
	return true;
}

This method first checks whether the sequence implements common collection interfaces, and, if so, retrieves the count directly. If it doesn’t, it manually enumerates that many elements from the enumerator to see if it runs out.

Using the enumerator directly is slightly faster than using LINQ methods; it avoids allocating an extra iterator state machine from Skip(). You can simplify it like this at a slight GC cost:

return sequence.Skip(count).Any();
// Or
using (var enumerator = sequence.GetEnumerator())
	return Enumerable.All(Enumerable.Repeat(0, count + 1), x => enumerator.MoveNext());

Each of these implementations involve one extra allocation (an iterator state machine) compared to the longer version.

Concurrency, part 4: Comparing promises frameworks in different languages

2015-01-08T00:00:00+00:00

Last time, I explained what promises are and how they work. Now, I’ll explore standard & third-party promise libraries in popular languages, explaining how they implement each facet of the promise paradigm.

.Net languages (C#, VB, etc…)

The .Net base class library includes promise classes called Task and Task<TResult>. These classes were introduced in .Net 4.0, released in 2010. Task is used for promises which have no value (simply running callbacks once the operation is complete); Task<TResult> inherits Task and exposes a value as well.

Methods

The then() method, which adds a callback to the promise, is called .ContinueWith(). It has a number of overloads which take different kinds of delegates, letting you pass callbacks that don’t return anything, return a synchronous value, or return a further promise for chaining. All of these methods return a new Task or Task<TResult> instance. The callback receives the original Task<T> object rather than the result; you can access the result using the Result property.
.ContinueWith() will, by default, run its callback even if the task failed. However, if the original task failed, accessing the Result property inside the callback will rethrow the exception, skipping the rest of the callback. You can add callbacks that only run on success or failure by passing TaskContinuationOptions.OnlyOnRanToCompletion or TaskContinuationOptions.OnlyOnFaulted.
The Deferred object used to create promises from existing asynchronous operations is called TaskCompletionSource<TResult>.
You can also create tasks from legacy APM (Asynchronous Programming Model, using IAsyncResult) using the Task.Factory.FromAsync() methods.
Promise creation helper methods include Task.FromResult() and Task.FromException() to create pre-fulfilled tasks, Task.Delay*( to create a task that will be resolved at a specified time in the future, and Task.WhenAll() and Task.WhenAny() to wait for multiple tasks.
You can run code on a background thread and get a promise of its eventual result by calling Task.Run().

Behavior

By default, .ContinueWith() will always run the callback asynchronously (using the current TaskScheduler), avoiding reentrancy bugs. You can override this behavior by passing TaskContinuationOptions.ExecuteSynchronously.

Tasks also support cancellation. Cancellation is a third fulfillment state that is mostly parallel to rejection. When creating a new Task from a TaskCompletionSource, you can call SetCancelled to set the resulting task to canceled. If a .ContinueWith() callback throws a TaskCancelledException or OperationCanceledException, the resulting task will also be canceled. Cancellation is designed to be used together with the CancellationSource system to cancel operations and callbacks.

Since .Net is a multi-threaded environment, you can also synchronously wait for task to finish by calling .Wait(). This will block the calling thread until the task is fulfilled, so it should not be called on a UI thread or in highly multi-threaded environments when threads are scarce.

Most confusingly, the Task class can also represent a delegate running on a background thread. You can create a new task from a delegate, which will return an unstarted task object. Calling Start() on this instance will run the delegate on its TaskScheduler and eventually resolve the task to its result. This use (and the entire Start() method) is confusing and should be avoided; instead, simply call Task.Run().

C# 5.0 & VB 11.0 took this one step further with language-level asynchrony using the await keyword, allowing you work with promises just like you would write synchronous code. Behind the scenes, the compiler transforms async code into a series of promise callbacks. Jon Skeet wrote a series of blog posts explaining exactly how this transformation works.

Availability

.Net 4.5 introduced *TaskAsync() versions of most of the framework’s non-CPU-bound operations, including stream IO, HTTP requests, database connections, and more. Note that the async methods in DbConnection & DbCommand will by default run synchronously and return a pre-completed task; it’s up to the derived connection clients to override these methods with non-blocking implementations. The built-in SqlConnection does override these methods; other ADO.Net providers may or may not.

All modern .Net unit testing frameworks have some level of support for async tests; for more details, see Stephen Cleary’s article on async .Net unit tests. Recent versions of ASP.Net MVC and Web API also allow async actions that return Tasks; ASP.Net vNext (6.0) also adds support for async action filters. Entity Framework also added support for async queries and updates (assuming the underlying provider implements async correctly) in version 6.

Outside Microsoft’s standard libraries, the situation is murkier. Older open source libraries for network-bound operations frequent lack async methods or implement them incorrectly (by running a blocking call on a thread pool thread instead of a non-blocking call). Newer libraries are more likely to get this right, especially with the introduction of language-level async support in 2012. Stephen Cleary wrote an excellent set of guidelines on how to properly write async APIs.

Java 7 – Guava

Java’s promise story is murkier. Java 1.5 introduced the Future<T> interface, which looks like a promise, but actually isn’t. A Future<T> does indeed represent a value which will be computed later, but it offers no way to notify upon completion – it has no then() method. All you can do with a future is synchronously wait for it to be completed. Thus, it is completely useless for non-blocking asynchronous operations.

To solve this, Google’s Guava library has its own promise-like interface called ListenableFuture<T>. This interface extends the standard Future<T> and adds a single addListener() method that takes a Runnable callback and an Executor to run the callback with (letting you control whether the callback will run synchronously or not). This is an extremely simple API designed to be easy to implement; the callback will run on both success and failure and does not allow chaining. Instead, the full richness & convenience of the usual promise APIs are available as static methods in the Futures class. x

Methods

By default, callback methods will run synchronously. You can make them run asynchronously by passing an Executor.

The then() method is Futures.transform(), which has overloads that take a callback returning a synchronous value or a further promise (if you don’t need to return a further promise, use Futures.addCallback()).
To handle errors, call Futures.withFallback(). This does not have an overload that accepts a promise-returning callback; if you need one, use Futures.dereference() to turn a ListenableFuture<ListenableFuture<T>> into a ListenableFuture<T>.
The Deferred object is SettableFuture<T>, which implements ListenableFuture<T> and adds mutation methods to resolve or reject the promise.
Promise creation helpers include Futures.immediateFuture() and Futures.immediateFailedFuture() to create pre-fulfilled promises, as well as Futures.allAsList() and Futures.anyAsList() to wait for multiple promises.
You run code on a background thread in a Java Executor and get a promise of the result by wrapping the executor in MoreExecutors.listeningDecorator() and calling submit()

For more information, see Guava’s introduction to ListenableFuture.

Java 8

Java 8 finally introduced a built-in promise interface, CompletionStage<T>.

Methods

Every method in CompetionStage has three variants: method(callback) runs its callback synchronously, methodAsync(callback) runs the callback asynchronously on the common ForkJoinPool, and methodAsync(callback, executor) runs the callback asynchronously on the specified executor.

The then() method is represented by a number of methods, depending on whether the callback consumes the value and/or returns a new value. thenAccept() consumes the value and returns void. thenApply() consumes the value and returns a different value. thenCompose() consumes the value and returns a new promise. thenRun() consumes nothing and returns nothing (it accepts a simple Runnable).
To handle errors, call exceptionally(), or handle() to run a callback whether the stage was resolved or rejected.
The Deferred object is CompletableFuture<T>, which implements CompletionStage (and is in fact the only built-in implementation), and adds mutator methods to resolve, reject, or cancel the promise.
Promise creation helpers are CompletableFuture.allOf(), CompletableFuture.anyOf(), and CompletableFuture.completedFuture().
CompletionStage also has binary versions of all() and any() as instance methods. Call stage1.{accept|applyTo|runAfter}Either(stage2, callback) (parallel to the standard then() variants listed above) to add a then() callback to whichever promise completes first. Call stage1.{thenAccept|runAfter}Both() to add a then() callback to both promises together.
You run code on a background thread in a Java Executor and get a promise of the result by calling CompletableFuture.runAsync() (returns a promise of void) or CompletableFuture.supplyAsync() (returns a promise of a value).

Availability

Unfortunately, CompletionStage is not used anywhere in the JDK standard library. There are third-party async libraries that return CompletionStages.

Javascript

Javascript has a much richer ecosystem of promises. Because Javascript is strictly single-threaded, JS programmers have had to deal with asynchrony since day 1. Different asynchronous APIs have taken different approaches to this; earlier browser APIs such as XmlHttpRequest uses events, and Node.js uses callbacks everywhere. Promises slowly began to arrive onto this landscape, including jQuery’s original version of $.Deferred or Closure’s goog.async.Deferred, which are mostly, but not entirely, like promises. Later came a variety of Node.js and/or browser-based promise libraries, including Q, Bluebird, and many others. Most recently, the upcoming ECMAScript 6 standard includes a Promise class (although many third-party implementations such as Q have richer APIs).

To simplify this mess, Kris Kowal, Domenic Denicola, and others put together the Promises/A+ spec, which codifies exactly how Javascript promises should behave. Promises/A+ specifies how promise resolution should work, and the exact signature and behavior of the then() method for adding success & error callbacks and chaining. It leaves everything else up to the individual implementations; most notably, it makes no mention of how to create a promise in the first place.

All modern promise implementations, including Q, Bluebird, Angular’s $q (which is based on Q), Bluebird, Closure’s goog.Promise, but with the notable exception of jQuery, including the ES6 promise class implement Promises/A+, and also add promise creation and convenience methods. The Promises/A+ site has a full list of conforming implementations.

Methods

The then() method is precisely specified by Promises/A+, and is thus (almost) the same in every implementation. then() accepts a success callback followed by a failure callback (both of which are optional), and will always call them asynchronously (on the next message loop iteration).
Many promise implementation add a third parameter to specify the this parameter with which to call the callbacks.
To add error callbacks, pass the second parameter to then. Most implementations also add a convenience method to only add an error callback; ES6 promises have .catch(); Q has .catch() or .fail(), and Closure promises have .thenCatch().
The Deferred objects vary between implementations. ES6 promises and Q 2.0 use a promise constructor as described previously. Q 1.0 has Q.defer(), which creates a Deferred object; Closure promises have goog.Promise.withResolver() which returns a similar object.
Promise creation helpers include Promise.resolve and Promise.reject to create pre-fulfilled promises, as well as Promise.all() and Promise.race() to consume multiple promises. These names are the same in most promise implementations.

For more details, consult the documentation for your specific implementation.

Availability

There are no standard APIs that return promises (yet). However, there are a large variety of open-source libraries (for Node.js and the browser) that expose promise-based APIs.

One of the advantages of the Promises/A+ standard is that all every standards-compliant implementation should be interoperable. You can take a promise from one implementation, and return it from a then() callback, or pass it to all() or race(), in a different implementation, and everything should work fine. This is particularly useful when consuming a library that uses a different promise implementation than your code uses.

jQuery Promises

jQuery introduced its own promise-like abstraction in version 1.5, $.deferred(). This feature was introduced before promises had a widespread standard API, and its API changed substantially in version 1.8 to align more closely with other promise libraries. jQuery still has one notable difference; promise callbacks are always executed synchronously (directly within resolve() or then()). jQuery deferreds also support the notion of a progress callback, which can be invoked multiple times before the promise is resolved to report incremental progress.

Methods

The then() method is then(), accepting optional success, failure, and progress callbacks and returning a chained promise of the callback’s return value.
Deferreds also have done() and fail() methods which add callbacks directly, but do not return a new promise. Instead, they return the original promise instance, like normal jQuery method chaining. This means that they can be used to add multiple callbacks to the same original value. However, the return value from the callback is completely ignored; to create further promises, you must use .then().
$.deferred() returns a mutable deferred object, which can be used to resolve the promise. Call .promise() on this object to get a readonly promise reflecting the result of the deferred.
jQuery has just one promise creation helper, $.when(). You can pass a single value to this method to create a resolved promise of the value. You can pass multiple jQuery promises (not other promise implementations) to this function to create a promise that waits for all of them. Note that you cannot pass an array of promises (if you do, you’ll get a resolved promise of that array itself); you must pass the promises individually. If you have an array, you can call $.when.apply(null, promisesArray) to pass each item in the array as a separate parameter.

Availability

jQuery returns promises from all of its asynchronous operations. $.ajax() (and equivalent helper methods like $.get(), $.getJSON(), or $.getScript()) return jqXHR objects which are promises with additional AJAX-related properties. jQuery objects also have a .promise() method which returns a promise of the end of the element(s)’s animation queue. This lets you easily wait for animations to finish running before continuing with other code via promise chains.

Next time: Advanced promise development guidelines

Concurrency, part 3: Promises – Asynchronous programming made easy

2015-01-05T00:00:00+00:00

Last time, I explored the two main options for writing asynchronous functions. Now, I’ll describe promises in more depth. The concept of a promise can be implemented in any language; the sample code is in Javascript, running in a Node-like environment. Part 4 will describe the details promise frameworks in various other languages.

Basics

A promise is an object that stores an asynchronously-computed value, or an error. The only way to consume the promise’s value (or error) is to pass it a callback by calling then() (some frameworks have different names for the method). A promise is always in one of three states: unresolved, fulfilled, or rejected. An unresolved promise has no value yet, and instead stores a list of callbacks to run once the promise is resolved. In this state, calling then() will store the callback in this list to be consumed later.

Once the operation backing the promise completes, the promise is resolved to a fulfilled or rejected state. At this point, the resolved value is stored in the promise, and the list of callbacks is executed and cleared. Once a promise has been resolved, it becomes completely immutable, storing only the result. Passing a callback to a resolved promise will execute it immediately. Some implementations will execute callbacks on resolved promises synchronously, making then() a potentially reentrant call; others, including all standards-compliant Javascript promises, will always run it asynchronously in a threadpool or a later message loop).

Note that a promise can only be resolved once. Thus, promises are only useful for operations that complete once and result in a single value (or collection). If you have an operation that can finish in multiple stages (eg, if it can have change events, or if different parts come in at different times), promises will not help; instead, you can use a stream abstraction, such as Node.js streams or .Net Reactive Extensions.

Creating promises

Promises are created from asynchronous operations. Modern asynchronous methods (eg, most new .Net APIs, and promise-aware Javascript libraries) will generally return promises themselves, letting you consume them without further effort. You can also create promises from other things, as described below.

Multi-threaded environments like C# or Java 8 also have thread pool APIs that run code in a background thread, then return a promise of the eventual result of the code. This is particularly useful in UI applications, to run CPU-bound work in a background thread, then consume the result in the UI.

You can also create promises manually. This is necessary when calling older asynchronous APIs that accept callbacks instead of returning promises. It can also be useful for more unusual asynchronous operations, such as creating a promise that will be resolved when a user clicks a button or closes a dialog, or waiting for specific commands to come in over a live network stream.

To create a promise manually, you need functions to call to manually resolve or reject the new promise, passing the resolved value or the rejection error. Your code would call these functions in the callback for the underlying operation, fulfilling the promise when the operation is finished. You then need to get the resulting promise object, which, while itself readonly, will reflect the fulfillment from your function calls.

Most promise frameworks provide this using a Deferred object (in .Net, this is called TaskCompletionSource), which is a mutable wrapper around a promise. The deferred object exposes its promise as a readonly property, and provides resolve and reject as member functions. Since deferred objects are mutable, they should not exposed publicly; instead, they should only be used to wrap the underlying asynchronous operation. The public interface (property or return value) should only expose the generated promise.

For example:

function readFilePromise(path) {
	var deferred = new Deferred();
	fs.readFile(path, function(err, data) {
		if (err)
			deferred.reject(err);
		else
			deferred.resolve(data);
	});
	return deferred.promise;
}

Modern Javascript promise implementations introduced a cleaner way to create a promise from an existing asynchronous method. You can create a new promise and pass a resolver callback. The promise constructor will this callback immediately, passing the resolve and reject functions as parameters. This makes scope management easier; instead of having a separate deferred variable, the mutable part of the promise is intrinsically scoped to the resolver callback. This may not work for more advanced operations, such as resolving a promise based on a UI event or a particular network response, but it’s a much nicer way to wrap existing callback-based functions.

For example:

function readFilePromise(path) {
	return new Promise(function(resolve, reject) {
		fs.readFile(path, function(err, data) {
			if (err)
				reject(err);
			else
				resolve(data);
		});
	});
}

More sophisticated promise libraries will also include helper methods to “promisify” existing async methods using standard callback patterns, such as Q’s nfcall & friends or C#’s TaskFactory.FromAsync overloads.

Finally, most promise frameworks include convenience methods to create pre-resolved or pre-rejected promises from existing values (useful when an async API has a synchronous “fast path” such as a cache), as well as promises that will be resolved after a given delay.

Composing promises

The biggest advantage to promise-based asynchrony is the ease of composition. You can return a value from a promise callback to get a new promise which will be resolved to that value once the original promise completes. This lets you easily create functions that compute values based on the result of an existing async operation, simply by returning the computed value from the promise callback, and returning the resulting promise from the function.

You can also return another promise from a promise callback, giving you a promise that will be resolved once this second operation finishes. This lets you chain together a sequence of asynchronous by simply returning the promise for each new operation from successive then() callbacks on the resulting promises. This is much nicer than nesting multiple completion callbacks from old-style async methods. In effect, this lets the inner promise “escape” from the scope of its promise callback, giving you a promise of its result in the original function.

For example:

var initialPromise = someAsyncFunction();
var delayedPromise = initialPromise.then(function(result) {
	return someOtherAsyncFunction(result);	// Returns a promise
});
// delayedPromise will be resolved with the 
// result of someOtherAsyncFunction()
var thirdPromise = delayedPromise.then(function(result) {
	return Object.keys(result).length;
});
// thirdPromise will be resolved with the
// number of properties after the other 2
// promises are resolved.

More formally, this feature is an example of a monad – a generic type that “amplifies” (adds more features to) any existing type. Other examples of monads include collection types, which amplify an existing type to store a number of values of that type, and optional/maybe types, which store either a None placeholder or a single value. The monadic unit operation, which creates a promise from an existing value, is Promise.resolve() (or equivalents in other frameworks). The mondaic bind operation, which applies a function to the value in a monad and returns a new monad containing the function’s result(s), is the then() method. This is what makes promises chainable. For a deeper introduction to monads, see Eric Lippert’s excellent series of blog posts.

Promise frameworks will also include helper methods to consume collections of other promises. Promise.all() takes a collection of promises and returns a new promise of an array of the results of these promises, thus waiting for all of the promises to be resolved. This can be used to kick off a number of asynchronous operations in parallel, then wait for them all to finish. Promise.any() takes a collection of promises and returns a promise of the result of the first promise to be resolved (ignoring all of the other promises).

For example:

function getWeather(city) {
	// Try 3 services and use whichever one answers first.
	return Promise.any([
		httpRequestPromise('https://weather1.example.com', { city: city }),
		httpRequestPromise('https://weather2.example.com', { city: city }),
		httpRequestPromise('https://weather3.example.com', { city: city })
	]);
}

function getTripWeather(itinerary) {
	return Promise.all(itinerary.cities.map(function(city) {
		return getWeather(city);
	}));
}

Error handling

Promises can also store error state. When an asynchronous operation fails, the resulting promise will be rejected rather than resolved, storing the error instead of the result. When this happens, none of the regular then() callbacks will run; instead, the promises returned by the then() calls will themselves be rejected. Thus, errors will automatically propagate down a promise chain; once an error occurs, the rest of the handlers will be skipped, until the error is handled. If an exception is thrown in any promise callback, the promise returned by that callback will also be rejected. Similarly, if a promise callback returns a promise (from a chained async operation) which is rejected, the next promise returned by then() will also be rejected.

To handle errors, you can add an error callback, typically either by passing a second callback to then() or by passing a callback to a separate function like fail(). These behave exactly the same as then() callbacks, except that they will only run when the promise is rejected. A resolved (as opposed to rejected) promise will skip error callbacks, just like a rejected promise skips normal callbacks.

Error callbacks will only handle errors from earlier in the promise chain. If you add both a success callback and an error callback in the same call (by passing two callbacks to then()), exactly one of the callbacks will run, depending on the fulfillment of the original promise. In contrast, if you call .then(callback).fail(otherCallback), and the original promise succeeds, but the first callback throws its own error, both callbacks will run (since you’re calling fail() on the rejected promise from the then() callback).

Error callbacks are still chainable; the promise returned by fail() will be resolved to whatever the callback returns. This means that once you add an error callback, the rest of the chain will keep running after an error occurs (error callbacks handle their errors). To prevent this, you can simply rethrow the error from the error callback, causing the next promise to be rejected again.

In effect, error callbacks behave exactly like catch blocks in imperative programming. Just like classical exceptions, once an error occurs (whether the original operation failed, or whether a callback threw any kind of error), the rest of the code (other callbacks) until the “catch block” (error callback) will be skipped. After the “catch block”, the rest of the code (further chained callbacks) will continue running, unless the “catch block” rethrows the error.

For example:

someFailingFunction()	// Returns a rejected promise
	.then(function(result) {
		// This code will not run
	})
	.fail(function(error) {
		console.log('Uh oh!', error);
		return 3;
	})
	.then(function(result) {
		console.log(result);	// Prints 3
		// Had the original operation not failed, this 
		// would have received the result of the first
		// then() callback.
	});

Unit testing

Asynchronous functions create special challenges for unit tests. A unit test for an asynchronous function must wait until the async part of the method finishes, and must catch exceptions or assertion failures that are raised in asynchronous continuations (in then() callbacks). A naive unit test that does not somehow wait for asynchronous operations to complete won’t actually test anything; if the asynchronous part fails, it will never be reported.

To solve this issue, most modern test runners allow unit tests to return promises. The test runner will check whether each test returns a promise, and, if it does, will wait for the promise to be resolved before concluding the test. If the returned promise is rejected, it will mark the test as failed, with the rejection reason as the failure message.

Therefore, you can call an asynchronous function in a test, add a then() callback that makes assertions about the result, and simply return that promise chain. If the assertions fail, the assert function will throw an exception, causing the promise returned by then() to fail, and failing the test.

For example:

function testSquareAsync() {
	return squareAsync(2)	// In real code, squaring a number should not be async
		.then(function(result) {
			assertEquals(4, result);
		});
}

To test that a method fails, you can add a success callback that throws an error and an error callback that handles the original error and asserts about it:

function testAjaxFailure() {
	return httpRequestPromise('https://cannot parse')
		.then(function(result) {
			assertFail('Expected failure but returned ' + result);
		}, function(error) {
			assertEquals(error, 'Bad URL');
		});
}

Next time: Comparing promises in different languages

Concurrency, part 2: Patterns for Asynchronous Methods

2015-01-04T00:00:00+00:00

Last time, I explained the basic concepts of asynchronous and multi-threaded programming. As I explained there, the most efficient way to run non-CPU-bound operations concurrently is to call them asynchronously. However, asynchronous code can be confusing and difficult to work with. This post will explain different techniques for writing asynchronous functions. The concepts described in this post apply to all languages; the sample code is in Javascript, running in a Node-like environment.

The problem

An asynchronous method cannot simply return its value like any other method. Since the result is computed asynchronously, the method will have already returned before the result arrives. In multi-threaded languages, this issue can be overcome by synchronously waiting for an asynchronous operation to finish. However, this completely defeats the point of asynchrony, which is to free up the thread while the operation is in progress. In single-threaded languages like Javascript, this is completely impossible, since the result cannot even arrive while the method is waiting.

Thus, an asynchronous method must use another approach to return its result. Asynchrony is viral; any method that calls an asynchronous method must itself become asynchronous. This effect will propagate up your entire call hierarchy, until the entry-point, which must either wait for the async operation (eg, Main() in C# or Java), or can ignore the operation’s result (eg, UI event handlers or network request handlers).

Continuation-passing style (callbacks)

The most basic approach to dealing with asynchronous operations is to use callbacks. With this approach, each asynchronous function call takes a callback parameter, which will be called when the operation completes. However, using callbacks in complex programs can be quite annoying. Performing multiple operations in sequence requires ugly nested callbacks. Every callback must explicitly check for and handle errors from the operation, with no equivalent of exceptions and catch blocks to handle control flow in error conditions. There is no way to store an asynchronously-retrieved value before it is ready.

In this approach, no function will ever return a value (the return statement is only used to exit a function early); instead, all execution flow between functions is handled using callbacks (to signal errors, return values, or just to wait for completion). This approach is called continuation-passing style.

This approach also relies on convention to tell the callback whether an error occurred. Node.js uses the convention that the callback’s first argument is always the error (or null if the operation succeeded); the result, if any, is passed as the second argument. Other approaches include passing the callback a status object with a (nullable) error property, or accepting two callbacks; one for success and one for error.

For example:

function addHash(path, callback) {
	fs.readFile(path, function(err, data) {
		if (err) return callback(err);
		var hash = computeHash(data);
		fs.writeFile(path + ".hash", hash, function(err) {
			if (err) return callback(err);
			console.log('Hash written');
			callback(null);	// No error
		});
	});
}

Promises

Promises represent a much better way to handle asynchronous operations. A promise is an object that represents a value that may arrive some time in the future. To consume the value, you call a then() method and pass a callback, which will run when the value arrives, or immediately if the promise has already been fulfilled. If this callback returns its own value, you will get a new promise of the resulting value, which you can then add more callbacks to later. Thus, you can chain asynchronous operations without nesting anything. If an error occurs, a promise can be resolved to an error state instead of a successful value, which will skip all success callbacks and instead return further rejected promises. Thus, errors will propagate naturally along a promise chain of asynchronous operations, until they are finally handled by error callbacks later in the chain. This is directly analogous to catch blocks in imperative programming. Finally, because promises are regular objects, they can be stored in fields just like any other value.

Using promises, the earlier example can be simplified to:

function addHash(path, callback) {
	return qfs.read(path)
		.then(function(data) {
			var hash = computeHash(data);
			return qfs.write(path + ".hash", hash);
		}).then(function() {
			console.log('Hash written');
		});
}

Next time: More about promises

Concurrency, part 1: Parallelism, Asynchrony, and Multi-threading Explained

2014-12-23T00:00:00+00:00

Concurrent programming techniques, such as multi-threading or asynchronous operations, are all the rage nowadays. As Moore’s law begins to fail, the industry is turning to concurrency to deliver the next generation of performance boosts. Every hot new development framework makes some kind of claim to run efficient asynchronous or multi-threaded code. With all of this hype, however, it’s easy to get confused about the exact meanings of “multi-threading”, “parallelism”, and “asynchrony”, and the difference between them.

About asynchrony

An asynchronous operation is an operation which continues in the background after being initiated, without forcing the caller to wait for it to finish before running other code.

When calling a typical, synchronous function, your code calls the function, and does not continue until the function is completely finished. If the function is CPU-bound (if all it does is compute values, without waiting for replies from anything else), this makes sense. The CPU is busy computing the result of the function, and doesn’t have time to do anything else (at least, on that thread).

However, if the function is not CPU-bound – if it simply waits for something else to finish – this makes less sense. If all the function does is wait for a response from a hard drive or database or web service, or even from a camera or other USB device, there is no need to keep the CPU busy waiting for that response instead of doing useful work. In that case, your entire program (or thread, in a multi-threaded environment) is stuck waiting for no reason.

This is why asynchronous operations exist. Instead of blocking the calling program (or thread) until a response arrives, an asynchronous (also called non-blocking) implementation will send of a request to the database or web service or whatever, then return immediately, letting your program continue running other code while the remote service sends a reply. Once the response arrives, the system will run a callback (either on your message loop or in a separate IO completion port thread, depending on the environment), letting your code handle the response.

About multi-threading

Multi-threading means running more than one thread of execution at a time. In this model, all operations are still synchronous, but the CPU will execute multiple threads of synchronous operations at the same time.

Multi-threading makes most sense when calling multiple (and independent) CPU-bound operations, on a multi-core processor. For example, a program that independently analyzes every pixel in an image could divide the image into one strip for each CPU core, then analyze each strip in its own thread at the same time.

Note that there is no advantage gained in running more threads than there are CPU cores. Threads do not magically let CPUs do more work for free; once you run out of dedicated cores to run your threads, you’ll end up with single cores running bits of each thread in turn, adding additional context-switching overhead as the core switches to each thread, and not getting any performance benefit.

How to best manage concurrent / parallel operations

Concurrency or parallelism simply mean running more than one operation at the same time. This can be accomplished using either asynchrony or multi-threading. However, different kinds of operations lend themselves towards different kinds of concurrency.

If you have strictly CPU-bound work, multi-threading is your only option. The whole point of asynchrony is to leave the CPU (and your thread(s)) free, so that you can run other code while waiting for the asynchronous operations to finish. If your work is CPU-bound, running it asynchronously does not make sense.

If you have non-CPU-bound work (disk IO, network requests, etc), both options can work. However, running the operations asynchronously will offer better performance. Parallelizing non-CPU-bound work using multiple threads means making each request on its own dedicated thread, blocking that thread until the response arrives. However, threads are not cheap; creating each thread occupies a megabyte or more of memory for the stack and limited kernel resources, and also adds more context-switching overhead. This places artificial limits on the total number of in-flight requests.

In contrast, asynchrony offers much better parallelism with non-blocking operations. You can kick off thousands of non-blocking requests from a single thread with very little per-request overhead, then schedule callbacks to run when each request finishes. Your original thread is then free to do whatever else it likes (such as actual CPU-bound work, kicking off more async requests, or accepting incoming connections).

Switching non-CPU-bound operations to non-blocking implementations can offer tremendous performance and scalability benefits, simply because threads are not cheap. At a previous job, I rewrote a long-running and massively parallel network server to use non-blocking IO, and the throughput improved from 900 connections per server to over 9,000 connections per server.

Handling UI threads

All of the discussion so far has been focused on raw performance for server-side or backend scenarios. However, asynchrony and multi-threading have one other, unrelated use: UI threads.

While your UI thread is busy (with any kind of operation), it will not accept any UI events, making your application hang and making users unhappy. Therefore, it is extremely important that all potentially long-running operations, whether CPU-bound or IO-bound, not run on the UI thread.

For CPU-bound operations, this simply means running them in a background thread; for non-CPU-bound operations, this can either mean running them in a blocking fashion in a background thread, or running them asynchronously from the UI thread. Unlike server scenarios, scalability usually isn’t a concern for UI applications (there generally aren’t thousands of operations to worry about), so it doesn’t actually matter much which approach you choose.

Availability in Programming Languages

As mentioned earlier, asynchrony and multi-threading are completely orthogonal concepts, and different languages expose different options.

Javascript is completely single-threaded (with the subtle exception of web workers, which still cannot share memory between threads), but also completely asynchronous. With very few exceptions (deprecated synchronous XmlHttpRequests in the browser, and the fs.*sync() in Node.js), every non-CPU-bound operation in Javascript is only available asynchronously.
This is why Node.js is so popular and scalable. Because everything runs asynchronously, as long as you don’t have any CPU-bound work, you can scale to thousands of concurrent requests with very little overhead (limited only by the memory consumed for each request).
Java, by contrast, is fully multi-threaded, with a powerful system of Executors and synchronization primitives, but has extremely little support for asynchronous operations. Before Java 8, Java had no built-in asynchronous operations, although there are some third-party libraries that offer async network IO.
C# is both asynchronous and multi-threaded. Like Java, it has a rich threading API, but unlike Java, it has had asynchronous IO methods since day 1. .Net 4.5 introduced simple-to-use asynchronous IO methods using the TPL and Task<T>, replacing the older, more cumbersome APM-based methods (Begin*() and End*()). However, outside the core .Net framework, many third-party libraries still do not offer asynchronous versions of IO-bound operations.

Next time: Patterns for Asynchronous Methods

MEF v2, Roslyn, and Visual Studio: An Adventure in Compatibility

2014-11-16T00:00:00+00:00

Background

Both Visual Studio and Roslyn use the Managed Extensibility Framework (MEF) to build large applications out of decoupled, extensible components. MEF allows different parts of these programs to talk to each-other using clearly defined interfaces, allowing different subsystems to be developed by different teams on different release cycles without breaking anything.

MEF is also used for extensibility. Visual Studio uses MEF to import services like syntax highlighting, IntelliSense providers, and other editor services; Roslyn uses MEF to import refactorings and diagnostics. This allows you to [Export] these services in your own extensions, and have Visual Studio or Roslyn automatically import and use them just by loading your DLL. This is the core of modern Visual Studio extensibility. Both Roslyn and Visual Studio also use MEF for hundreds of internal services, to help wire up core functionality even without being extensible. The core compilation engine (including the syntax trees and semantic model) do not use MEF at all, but the workspaces layer uses it for services like caching layers, as well as formatting rules and language-specific APIs to create core concepts like compilations. The Visual Studio & editor integration layers use hundreds of their own MEF services to wire up their own implementations.

The Problem

The core parts of Roslyn (the compilers & syntax / semantic APIs, and the basic workspace services) were recently changed to be portable assemblies, allowing them to be used in Windows Store apps as well. The original version of MEF (System.ComponentModel.Composition.dll) is not a portable assembly, so the Roslyn team switched to MEF v2 instead. MEF v2 is released as a NuGet package named Microsoft.Composition, which contains portable (PCL) assemblies that solve all of these problems. You can see more details about the change in this commit (this commit broke Roslyn’s CodePlex sync tool, so you can only see the files changed on GitHub or a local clone).

However, Visual Studio itself cannot move away from MEF v1, because that would break every existing extension that exports any service using the MEF v1 [Export] attribute. Roslyn’s Visual Studio integration (especially editor services like syntax highlighting) need to export VS editor services (eg, IClassifierProvider) to Visual Studio’s MEF host, while at the same time importing Roslyn workspace services which are exported using MEFv2, and never the twain shall meet.

The Solution

The Visual Studio 2015 Preview is the first public VS build to suffer from the problem (Dev14 CTP 4 was built before Roslyn switched to MEFv2). To fix it, Visual Studio has its own internal version of MEF (internally referred to as MEFv3 or usually VsMEF), contained in Microsoft.VisualStudio.Composition.dll. This version of MEF scans assemblies for both MEFv1 exports and MEFv2 exports, loading them all into a single unified container.

This solves all of the problems. This new MEF container picks up MEFv1-exported types from existing extensions and from the rest of Visual Studio, and can seamlessly mix them with MEFv2-exported types from Roslyn. As long as you don’t mix attributes from different MEF versions in the same class, everything will just work. This change is so transparent that extension developers won’t even notice it unless they go out and look for it; I only discovered it because it completely broke my out-of-process VS MEF host in VSEmbed.

This also means that you can use MEFv2 instead of MEFv1 in VS2015 extensions. Note that older versions of Visual Studio will completely ignore MEFv2 attributes. You can utilize this to create parts that will only be imported by VS2015, by using the MEFv2 [Export] attribute on only those parts in a backwards-compatible extension. Note, however, that you can’t use any new types or assemblies in class members, or older versions will fail to load them as MEFv1 tries to scan every type for its attributes.

VS MEF also has other benefits, including faster loading and better error messages when things go wrong.

VS MEF will hopefully be published standalone on NuGet some time in the future.

Getting Started with the Visual Studio Theming Architecture

2014-09-04T00:00:00+00:00

Visual Studio 2010 rewrote the entire shell UI – the MDI tabs & tool windows, the toolbars, and the menus – in WPF. This let it use WPF’s powerful resources / theming system to style all of the UI elements, which Microsoft took advantage of in VS 2012 to add multiple themes (Light, Dark, and, later, Blue). This blog post will explore how this system is implemented. Later blog posts will explain how to use VS theme colors & controls in your own VS extensions, and, later, how to use VS theming directly in your own standalone application (which will require an installed copy of VS to run).

The theming system is built on WPF ResourceDictionaries, which contain reusable global resources such as colors, brushes, and control styles, keyed by strings or objects. Visual Studio 2010 had a table of 463 theme colors in the Microsoft.VisualStudio.Shell.VsColors class (and VsBrushes that return brushes instead of colors), containing every color that used by VS2010’s new UI theme. Visual Studio 2012 then greatly expanded the theming system, theming the window chrome and introducing multiple color themes. To power this, it replaced this table of theme colors with Microsoft.VisualStudio.Shell.ThemeResourceKey, which contains a category and an entry name, and indicates whether it’s specifying the foreground or background color for that entry, and whether it refers to a color or a brush. For compatibility, the older VsColors & VsBrushes still exist in later versions of VS as well, bound to the same colors as the corresponding ThemeResourceKeys in EnvironmentColors.

ThemeResourceKeys are broken down by category and contained by a number of classes (mostly) in Microsoft.VisualStudio.PlatformUI, including EnvironmentColors, CommonControlsColors, BackstageColors, TreeViewColors, StartPageColors, and more (some of these were introduced in VS2013 or Dev14). For a complete list, open Microsoft.VisualStudio.Shell.XX.0 and Microsoft.VisualStudio.Shell.UI.Internal (beware that this is unversioned, so it’s trickier to use in cross-version extensions) in your favorite decompiler, then search for usages of ThemeResourceKey (which can be found in Microsoft.VisualStudio.Shell.Immutable.11.0). Note that ThemeResourceKeys are compared by value, so you can use keys from later VS versions without referencing their DLLs by copying the key declaration into your own code. Obviously, such keys will not have any colors in earlier VS versions, but this lets you use them without causing compilation issues (as long as you check the runtime VS version before using them). Note also that any keys declared in unversioned DLLs are not guaranteed to exist in future VS versions either.

These keys (both the strings from VsColors & VsBrushes and the ThemeResourceKeys from later versions) are added to a global ResourceDictionary in System.Windows.Application.Current by the internal ResourceSynchronizer class when VS is launched, allowing them to be referenced in any WPF control anywhere in Visual Studio (in VS2012 and later, the same class will also repopulate this ResourceDictionary whenever you change the current theme). This lets you easily use theme colors in your own extensions.

To use theme colors in your own VS extension, you simply need to add a reference to Microsoft.VisualStudio.Shell.XX.0 (from the minimum VS version that you want to support) and declare it in your XAML. You can then use {DynamicResource}s with {x:Static} declarations to use the keys, and they will pick up the correct theme colors at runtime. For example:

<Control x:class="YourAddin.YourControl"
		 xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
		 xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
		 xmlns:shell="clr-namespace:Microsoft.VisualStudio.Shell;assembly=Microsoft.VisualStudio.Shell.10.0">
	<Border x:Name="NotificationWindow" Loaded="NotificationWindow_Loaded"
			Background="{DynamicResource {x:Static shell:VsBrushes.CommandBarOptionsBackgroundKey}}"
			TextElement.Foreground="{DynamicResource {x:Static shell:VsBrushes.CommandBarTextActiveKey}}"
			BorderBrush="{DynamicResource {x:Static shell:VsBrushes.DropDownBorderKey}}"
			BorderThickness="1">
		...
	</Border>
</Control>

If you aren’t targetting VS2010, you can also use other resources such as CommonControlsColors inside the Microsoft.VisualStudio.PlatformUI namespace. If you need colors rather than brushes (for example, if you’re constructing a gradient), use VsColors instead of VsBrushes.

Although this will work correctly at runtime within Visual Studio, you won’t see any colors in the XAML designer, since it doesn’t load any VS themes. In part three of this series, I’ll explain how to load a VS theme into the XAML designer yourself, so you can make sure your colors match without having to build & run your extension after every change.

Next time: Escaping the Box: Using VS themes outside Visual Studio

How to add a menu item to a Visual Studio extension

2014-08-26T00:00:00+00:00

As I’ve described earlier, when creating a Visual Studio extension, you are given a choice between the older VsPackage-style extension (using a wizard from the New Project dialog) and the newer, MEF-based extensions.

Newer parts of VS, such as the editor, the Web Tools, or Roslyn, are built using MEF, so an extension simply needs to export MEF services for these components to import & run. These MEF-based extensions are much simpler and easier to work with. However, if you create an MEF-based extension (eg, from a Roslyn or editor extension template), there is no obvious way to add a menu command. This blog post explains how to do that.

You can follow along in your own extension, or you can see all of these changes in this commit in Ref12.

Visual Studio menu commands are compiled from .vsct files, which are compiled by the VSCT compiler into embedded .ctmenu resources within the DLL. To get started, you need to add a .vsct file (the name doesn’t matter) to your project as a <VSCTCompile> tag. This must be done by editing the project file directly. You’ll also need to add an empty VSPackage.resx file for the VSCT compiler to embed the compiled resource into

The project file modifications (add these into any <ItemGroup>) are as follows:

	<VSCTCompile Include="YourName.vsct">
		<ResourceName>Menus.ctmenu</ResourceName>
	</VSCTCompile>
	<EmbeddedResource Include="VSPackage.resx">
		<MergeWithCTO>true</MergeWithCTO>
		<ManifestResourceName>VSPackage</ManifestResourceName>
	</EmbeddedResource>

The contents of the VSCT should look like this:

<?xml version="1.0" encoding="utf-8"?>
<CommandTable xmlns="http://schemas.microsoft.com/VisualStudio/2005-10-18/CommandTable" xmlns:xs="https://www.w3.org/2001/XMLSchema">
	<Extern href="stdidcmd.h"/>
	<Extern href="vsshlids.h"/>
	<Commands package="PackageGuid">
		<Buttons>
			<Button guid="CommandsGuid" id="MyCommand" priority="0x0101" type="Button">
			<Parent guid="guidSHLMainMenu" id="IDG_VS_CODEWIN_NAVIGATETOLOCATION"/>
			<Icon guid="guidObjectBrowserButtons" id="15"/>

			<Strings>
				<ButtonText>My &amp;Powerful Command</ButtonText>
			</Strings>
			</Button>
		</Buttons>
	</Commands>

	<Symbols>
		<GuidSymbol name="PackageGuid" value="«Package GUID»" />

		<GuidSymbol name="CommandsGuid" value="«Command Group GUID»">
			<IDSymbol name="MyCommand" value="0" />
		</GuidSymbol>
	</Symbols>
</CommandTable>

The command ID values must be unique within each command group (GUID), so you don’t need to worry about colliding with other commands.

The <Parent> entry specifies the group that the command should be placed in; you can find all standard groups in Program Files\Microsoft Visual Studio 12.0\VSSDK\VisualStudioIntegration\Common\Inc\vsshlids.h. You can also find a list of all command IDs in stdidcmd.h in the same folder.

2. Define a VSPackage for Visual Studio to load

To make Visual Studio load your menu command file, you need to define a VSPackage within your addin and register it in the VSIX.

First, add the following code file (the class name does not matter):

using System;
using System.Runtime.InteropServices;
using Microsoft.VisualStudio.Shell;

namespace YourAddin.Namespace {
	// These GUIDS and command IDs must match the VSCT.
	[ProvideMenuResource("Menus.ctmenu", 1)]
	[Guid("«Package GUID»")]
	[PackageRegistration(UseManagedResourcesOnly = true)]
	class YourAddinPackage : Package {
	}

	[Guid("«Command Group GUID»")]
	enum YourAddinCommand {
		MyCommand = 0
	}
}

The attributes in the package class tell CreatePkgDef.exe (which is invoked by the VSIX compiler) how to emit a .pkgdef file in your addin that sets he appropriate registry keys to register the addin and its menu resource with the Visual Studio’s native COM-based addin system. This is completely independent of MEF registration for managed addins.

The enum and its [Guid] attribute are simply a convenient way of keeping track of the GUID and command IDs; you’re free to use a different approach if you like.

You also need to tell the VSIX to include the package and generate the pkgdef file by adding the following line to the <Content> element in your source.extension.vsixmanifest:

		<VsPackage>|%CurrentProject%;PkgdefProjectOutputGroup|</VsPackage>

If you’re using the designer (for version 2 VSIXes), go to the Assets section, click New, select Microsoft.VisualStudio.VsPackage, A project in the current solution, and select your project.

3. Fix the project configuration

Finally, you need to make a few changes to the csproj file to correctly register the addin:

First, delete <GeneratePkgDefFile>false</GeneratePkgDefFile> if it’s present to tell the VSIX compiler to generate a pkgdef to register your package. Purely MEF-based addins don’t need pkgdef files at all, so most project templates will add that, but once you create a VSPackage, you need a pkgdef to register it with Visual Studio’s native COM addin loader.

Next, add the following markup anywhere in the root element (not in an <ItemGroup>):

	<PropertyGroup>
		<UseCodebase>true</UseCodebase>
	</PropertyGroup>

VSPackages are actually COM DLLs that are registered in the Visual Studio hive’s config key and loaded using normal COM practices. Managed VSPackages are loaded using COM interop; they are registered as mscoree.dll, which then looks up a managed assembly to load from the same registry key. By default, the PkgDef creator will emit "Assembly"="YourAssemblyName", which mscoree will try to load using the standard .Net assembly loader. Unless your VSPackage DLL is in the GAC (which it won’t be), this won’t work, and VS will silently refuse to load your package. Setting <UseCodebase> in the project file will make it emit "CodeBase"="$PackageFolder$\YourAddin.dll" ( $PackageFolder$ is substituted for the actual DLL path when the VSIX is installed), which tells mscoree the exact path to load it from.

4. Implement the command

After doing all this, your command should show up when you hit F5 in the addin (running in the Experimental instance). However, you still need to actually implement the command so that something will happen when the button is clicked.

For global commands, the simplest way to do this is to override the Initialize() method in your package and add code like the following:

var mcs = (OleMenuCommandService) GetService(typeof(IMenuCommandService));
var commandId = new CommandID(typeof(YourAddinCommand).GUID, (int)YourAddinCommand.MyCommand));
mcs.AddCommand(new MenuCommand(delegate {
	MessageBox.Show("You clicked me!");
}, commandId);

For commands that apply to the active editor window, I wrote a class which adds an interceptor to the editor’s command handler chain, and exposes a simple interface that accepts an entry in a command enum. This can be used both to implement new commands and to intercept existing ones. You can find it at CommandTargetBase.cs, and you can see sample usage here.

Exploring Roslyn, part 3: Breaking Changes

2014-05-28T00:00:00+00:00

Last time, I talked about what the Roslyn End-User Preview does and how it works. This time, I’ll talk about some subtle breaking changes in the compiler.

The C# & VB teams have a very high bar for backwards compatibility; they try extremely hard to make sure that all of your existing code will compile identically in Roslyn. However, in any project as large as a compiler, there are bound to be some changes in the way compilation works. Some of these are flaws in the old compiler, for which Roslyn’s fixes can subtly break existing code; some of these are side-effects of the way the new pipeline works which were deemed not worth fixing. All of these changes are in extremely subtle corner cases, and should not affect most normal codebases.

Breaking changes come in two varieties: Forward-breaking changes, meaning that code that worked pre-Roslyn doesn’t work with Roslyn, and backward-breaking changes, meaning that code that works with the Roslyn compiler doesn’t work with the old compiler (these can cause problems with open source projects if different people use different VS versions).

Obviously, all of the new language features in C# 6 are backwards-breaking changes; they will never work with older compilers. However, there are some more subtle changes which can prevent projects from being compiled without Roslyn even if they don’t use any new features.

Lifting variables in iterator methods

When you write an iterator method (using yield return), the compiler will transform your method into a class which implements IEnumerable<T>, turning your actual code into a state machine inside MoveNext(). (See Jon Skeet’s blog post for a thorough explanation.)

Local variables in your method become fields in the iterator class, so they can be persisted across calls to MoveNext(). The old compiler would always lift every single local variable to a field, even if it’s never used across a yield return boundary (meaning that it doesn’t need to stick around between MoveNext() calls). The Roslyn compiler is smarter, and only lifts locals when it needs to (if the local is used both before and after a yield return).

This is a breaking change in both directions. If your pre-Roslyn code declares something like a timer or file handle in an iterator (and doesn’t use it later), which it expects to survive until the iterator itself is collected, compiling with Roslyn will allow it to be GC’d much earlier that it would have been with the old compiler. To fix this, simply add a GC.KeepAlive() call at the bottom of the iterator. Since it is now used on both sides of the yield return, the compiler will be forced to lift it into a field in the iterator.

This can also be a backwards-breaking change, for two reasons. Most obviously, if your Roslyn code uses a very large object in only one place in an iterator, compiling it with the native compiler will cause that object to last much longer; potentially leaking large amounts of memory if your iterators are long-lived (which can happen in surprising ways with LINQ queries).

More subtly, this means that code compiled with the native compiler can load types earlier than it would have with Roslyn. This bit me in Ref12, where I had an iterator that used a type in an unversioned assembly that I loaded using a AssemblyResolve handler. With Roslyn, it worked fine. Without Roslyn, that type ended up in a field in the iterator class, so MEF tried to load the type when inspecting the assembly (through Assembly.GetTypes()), and the extension refused to load.

Demo

static void Main() {
	var e = M();
	e.MoveNext();
	GC.Collect();
	GC.WaitForPendingFinalizers();
	
	e.MoveNext();
	GC.Collect();
	GC.WaitForPendingFinalizers();
}

static IEnumerator M() {
	var a = new Alerter();
	a.ToString();	// Prevent the optimizer from removing the variable entirely
	Console.WriteLine("Before return");
	yield return null;
	Console.WriteLine("After return");	
}
class Alerter {
	~Alerter() { Console.WriteLine("Destroying!"); }
}

When compiled with the native compiler, this code will print

Before return
After return
Destroying!

Since the Alerter instance is stored as a field in the iterator, it cannot be collected until the iterator is finished.

With Roslyn, it prints

Before return
Destroying!
After return

Since the variable is not referenced again after yield return, Roslyn compiles it to a local variable within MoveNext(), allowing it to be GC’d while the iterator is still alive.

Loading indirectly referenced assemblies during compilation

This is a backwards-breaking change only; it will not break any code that already works with the pre-Roslyn compiler.

When compiling code that uses types from a different assembly, the compiler needs to load that assembly to look up the metadata for those types. It will also load assemblies for types that are related to methods you call directly, such as base types, or parameter types in other overloads.

The native compiler took a heavy-handed approach; it always tried to load every base type and interface for any type that you use. The Roslyn compiler is more frugal; it will only load base types and interfaces if it needs to.

This means that projects created with Roslyn can easily be missing references that will prevent them from compiling with the native compiler. In the worst case, there are types like ExtensionManagerService (in Visual Studio) that implement COM interop interfaces in private PIAs (Microsoft.Internal.VisualStudio.Shell.Interop.11.0.DesignTime.dll) that are not available outside Microsoft. With Roslyn, you won’t even notice these; the IDE does not show them, and the compiler completely ignores them. The native compiler, on the other hand, will completely refuse to compile code that uses these types, since it can’t load those base interfaces. This bit me here. (The solution is to create your own PIA assembly from the embedded interop types in ILSpy.)

Overload resolution with conflicting ambiguities

C# 6.0 has some subtle changes in the behavior of overload resolution with optional parameters. Before Roslyn, if you had a method call that was ambiguous between two overloads, but one of those overloads had additional optional parameters, whereas the other didn’t, the native compiler would choose the first overload.

For example:

void M(string a) { Console.WriteLine(1); }
void M(Exception a, char c = 'c') { Console.WriteLine(2); }

M(null);	// Prints 1, not 2

In Roslyn, this will no longer compile; the compiler will instead complain that it can’t figure out which overload to pick. You can fix this by explicitly casting the initial parameter to the type used in either overload.

Overload resolution is extremely complicated, but Roslyn’s behavior appears to be correct according to the spec (§7.5.3.2; emphasis added):

For the purposes of determining the better function member, a stripped-down argument list A is constructed containing just the argument expressions themselves in the order they appear in the original argument list.

Parameter lists for each of the candidate function members are constructed in the following way:

The expanded form is used if the function member was applicable only in the expanded form.

Optional parameters with no corresponding arguments are removed from the parameter list

After removing omitted optional parameters, it is clear that neither of these overloads is better, so the ambiguity error is correct.

Other bugfixes

The Roslyn compiler also fixes a number of minor bugs in the old compiler, rejecting code that was incorrectly accepted by the native compiler (such as the overload resolution example above), or accepting code that the old compiler incorrectly reported an error for.

Roslyn will show a warning when comparing a non-null value type to null (eg, new DateTime() == null).
The native compiler missed this warning for non-primitives (even though the optimizer collapsed it to a constant false).
Roslyn does not allow is or as to be used with static types (eg, new object() is BitConverter).
The native compiler compiled this fine (it always evaluated to false at runtime).
Roslyn allows structs to contain themselves as static nullable fields (eg, struct MyStruct { static MyStruct? Value; }). The native compiler incorrectly complained about a cycle in the struct layout.
Note that there is currently a bug in Roslyn that makes this produce invalid IL.
Roslyn rejects parentheses around named parameters (eg, Console.WriteLine((value: 42))).
The native compiler incorrectly allowed this invalid syntax.
Roslyn disallows method groups in is or as operators (eg, "".Any is byte).
The native compiler allowed this (although it did show a warning), and in fact didn’t even check extension method compatibility, allowing crazy things like 1.Any is string or IDisposable.Dispose is object. Interestingly, the spec explicitly allowed this (§7.10.10); the Roslyn team felt that that was not a good idea.

Exploring Roslyn, part 2: Inside the End-User Preview

2014-05-21T00:00:00+00:00

Last time, I described the basics of the Roslyn compiler platform, including how the different layers are built and interact with each-other. In this post, I’ll talk about what the End-User Preview does and how it works.

The Roslyn End-User Preview provides the first peek at the new Visual Studio managed editing & debugging experience, powered by Roslyn. The preview completely replaces all of the native language services (the old DLLs will not be loaded at all) with the new Roslyn-powered versions. These new DLLs are written entirely in C# and VB (except for a small C++ layer, used to hack around COM interop limitations when talking to other parts of VS).

As you can imagine, replacing the native language services is a gargantuan task, especially since the team is maintaining its usual commitment of not breaking muscle memory. All of the editing behaviors you’re used to, including snippets, IntelliSense preselection & commit patterns, and more, work as they did before, so your accustomed patterns of keyboard shortcuts will continue to work. To make sure this is done perfectly, the Roslyn team wants your feedback. Take out some time, download the preview, and enjoy the enhanced editor experience. If you notice anything that doesn’t behave the way it used to, take out a but of time and file a bug.

What’s in the preview?

The preview has two components: the new Roslyn compilers, which replace the native compilers when building your projects, and the language services, which power the editor & debugger within Visual Studio. Although these components sounds unrelated, they will not work in isolation; the debugger (in particular) makes very specific assumptions about the IL generated by the compiler, and won’t work properly with the wrong version of the compiler.

Compilers

The compiler preview is not hosted by Visual Studio. Instead, the preview extension will edit the per-user MSBuild targets file (see source) and replace the definition of the native Csc and Vbc tasks with the new Roslyn-based versions (Csc and Vbc).

Note: This means that as soon as the Roslyn Preview package is loaded (when you create or open a C# or VB project), in any Visual Studio hive, the Roslyn compilers will become active for all MSBuild v12 builds. If you want to build anything using the native compilers, you can either use an earlier version of MSBuild (eg, the VS 2012 command prompt), or set the DisableRoslyn environment variable to true. To check whether you’re running the Roslyn compiler, look for VBCSCompiler.exe in Task Manager.
The command-line vbc.exe and csc.exe are not affected. (since this just modifies the MSBuild task definition)

The compiler preview uses an interesting approach to maximize performance. Because it’s implemented as a per-user VS extension, the compiler assemblies cannot be NGen’d (since that requires that the assemblies be in the GAC, which needs admin privileges and is machine-wide). Therefore, every time you start the compiler, the JITter must re-compile all of the compiler’s IL to machine code. Since the compilers are enormously complicated, that would be a noticeable performance hit. To avoid this delay, the actual compiler code runs in a shared compilation server process called VBCSCompiler. When you first compile something (whether directly rcsc and rvbc command-line compilers, or whether through the MSBuild tasks), it will launch this compilation server (thus JITting all of the compilation code only once), then communicate over a named pipe (source) to ask it to compile your project. The server will then sit in the background until you next compile something, which it will be able to do instantly. Therefore, you may notice a delay the first time you compile something after rebooting your machine.

This hack is only necessary because the preview is a per-user install. The next release of Visual Studio will include the Roslyn-powered compilers as part of the machine-wide install of MSBuild v14, which will be properly NGen’d and won’t need to do any of this.

Language services

The second part of the preview is the new language services. This part replaces the native language services packages with new Roslyn-powered versions. Since this lives entirely within Visual Studio, the preview doesn’t need any ugly hacks to register itself. The only difference between the preview package and the built-in Roslyn-based language services that will be released with Dev14 is that the preview package also disables the VS package for the native language services (in the VS registry hive). Unlike the compilers, the language service has no effect outside the hive it was installed in.

The Roslyn-based language services faithfully reproduce the complete development experience from the native language services (except a couple of features that have not yet been implemented), as well as a host of new features that were made possible by the newly well-organized codebase. These new editor features fall into a couple of categories:

Syntax Highlighting

Roslyn extends the editor’s existing syntax highlighting to a number of new areas. Tooltips for types and method signatures now syntax-highlight the declarations, including keywords and separate colors for User Types (classes, structs, etc), if configured in Fonts & Colors options (the Productivity Power Tools extension also has this feature, but Roslyn’s works better). It will even show syntax highlighting for members referenced in <see /> tags in the doc comments!

Tooltips for collapsed methods or regions now show a fully syntax-highlighted preview of the code in the block.

References to members in <see>, <param>, <typeparam>, and <exception> when editing XML doc comments tags are now syntax highlighted, and participate in identifier highlighting.

The language service will also identify unnecessary qualifiers in your code (already-imported namespaces, inferable generic parameters, unnecessary casts, this., etc) and dim them in the editor to show that they can be removed. (there is also a refactoring to automatically remove them)

IntelliSense

Roslyn also features a vastly improved IntelliSense engine. The editor now knows exactly what keywords can appear in every context, giving you accurate IntelliSense for areas like property declaration blocks, class modifier keywords, preprocessor directives, and many other areas that the old C# language services fell short.

The Roslyn editors also add IntelliSense for <see /> (and similar) tags in XML doc comments, allowing you to select any type or member and insert the correct doc comment ID (with generics declared appropriately). This makes it far more convenient to write correctly-structured XML comments, which is particularly useful when compiling them to documentation sites (eg, with Sandcastle).

Refactorings

The crown jewel of the new language services is the all-new refactoring engine. All of the existing refactorings have been reworked using the new semantic APIs to preserve the meaning of your code, even if you introduce conflicting names or other challenges. To make this work, the refactoring engine checks which symbol every identifier resolves to, then makes sure that each one still refers to the same symbol after the rename is complete. If it detects any changes, it will fully-qualify the offending reference to force it to refer to the correct (original) symbol (it will then reduce the reference to become as simple as possible).

The rename refactoring in particular has been completely redone. Instead of showing a modal dialog asking for a new name, the Roslyn-powered rename experience is completely inline. After pressing F2, you can start typing the new name directly in the editor, and all references to the symbol you’re renaming will be updated in real-time, including surrounding changes to resolve conflicts. (unresolvable conflicts will be highlighted in red)

The refactorings (both old and new) are now exposed contextually. Press Ctrl + Dot to see which refactorings are available for the current cursor or selection in a smart tag popup; the language service will even show a syntax-highlighted preview of the generated code for each action.

The existing refactorings include

Generate variable/type/method/enum/constructor
Fully qualify reference/Add import
Remove unused usings
Implement interface/abstract class

As well as the existing VB-specific syntactic fixes:

Add End statement
Fix Exit / Continue loop keyword
Move Option statement to top of file
Identifier spell check

There are also a number of completely new refactorings:

Simplify type name (removes unnecessary suffixes or namespace qualifiers, and uses keywords or using aliases where applicable)
Remove unnecessary cast
Add missing assembly reference
Fix function return type (VB only)

Next time: Breaking Changes

Exploring Roslyn, part 1: Introduction

2014-04-07T00:00:00+00:00

The .Net Compiler Platform, codenamed “Roslyn”, is the most ambitious project from Microsoft’s Developer Division in recent years. The C# and VB teams got together and rewrote the compilers and language services from scratch in managed code (a mix of C# and VB), replacing a horrible mess of C++ code (with some managed code) that had mutated over the past ten years into a complex and difficult-to-modify codebase. Now, after over five years of work, Roslyn is almost finished, and is an integral part of Visual Studio 2015, now available in preview.

Why rewrite everything?

The native compilers were so complicated that even simple language changes required enormous amounts of work. The new managed compilers have a much better design which makes changes easy. (note that actually designing a good language feature still requires a great deal of design work)
The native compilers were completely closed, so that any other component that needed to parse C# or VB source (eg, the editor, the Razor compiler, and certain Visual Studio services) each required separate parsers for C# and VB.
Because C# and VB have different histories and origins, the implementations of the native compilers and IDE services were completely different, involving a lot of logic that needed to be written (and maintained) twice for no good reason. This also led to lots of pointless inconsistencies in the way the editor behaves (eg, code snippets).
Because the compilers didn’t expose any information about what they were compiling, implementing VS refactoring tools required a great deal of work to reproduce the semantic meaning of the code being refactored. This is the primary reason that VS has so few refactorings, and why they’ve barely changed in nearly ten years.

What’s so cool about Roslyn?

Roslyn is completely open source! You can browse the complete source code for the compiler toolchain and Visual Studio integration at source.roslyn.io, and you can even make your own contributions on GitHub.
Roslyn exposes the compiler’s syntax tree and semantic model, allowing you to easily write C# (or VB) code that parses a source file and explores – or even rewrites – the code. This makes it easy to build all sorts of tools to analyze or modify source code.
Roslyn exposes a simple extensibility model that lets you create your own warnings or refactorings that integrate seamlessly with Visual Studio.
Thanks to the newly accessible semantic information, Roslyn includes a number of new refactorings, and has vastly improved the existing ones (especially Rename).

Inside Roslyn

Roslyn has a number of layers, each of which consumes the previous layer and adds more features. As of early February 2015, the entire Roslyn stack, including the Visual Studio & debugger integration layers, is open source on GitHub.

Core Compiler Models

Microsoft.CodeAnalysis.dll contains the basic types and infrastructure used by both compilers, including the base types for the syntax and semantic trees, a large collection of utility classes, and the code-generation infrastructure. This includes all parts of the compilation & analysis processes that are not language-specific.

This layer, as well as the next three layers, is split into pairs of portable and desktop assemblies. The portable versions are full PCL assemblies that can run on platforms that do not offer standard filesystem access, such as Windows Runtime or Windows Phone. The .Desktop variants add filesystem-aware APIs, and can only run on full .Net platforms (including Mono). For example, the portable assemblies will compile and expose syntax trees & semantic models, and the Desktop layer adds the ability to resolve referenced DLLs from disk.

Language-specific compilation code

Microsoft.CodeAnalysis.CSharp.dll and Microsoft.CodeAnalysis.VisualBasic.dll contain the concrete implementations of the VB & C# compilers, building on top of Microsoft.CodeAnalysis.dll to implement language-specific logic. These assemblies include parsers, concrete syntax and semantic tree types, and all of the actual compilation logic for their respective languages.

These assemblies are invoked by the compiler applications to actually compile code, and can be used directly to parse & explore the syntax or semantic trees.

Actual compilers

This layer includes the applications that you actually invoke to compile code, including csc.exe and vbc.exe (the Roslyn versions of the command-line compilers), Microsoft.Build.Tasks.Roslyn.dll (the MSBuild task invoked from project files), and VBCSCompiler.exe (the compilation server process that actually runs the compilation code for the MSBuild task to avoid JIT delays; see part 2).

These projects are surprisingly small. Since all of the actual work happens in the previous layer, all they do is parse the command-line options, then invoke the compilation DLLs. (The MSBuild task is more complicated, since it needs to launch & communicate with the compilation server)

Workspaces

The previous layers take a very localized view of C# and VB code. They don’t know anything about Visual Studio, or even project files; instead, they operate on raw source code, bundled in a Compilation object that keeps track of all of the code being compiled, as well as compiler-level options like referenced assemblies.

The Workspaces layer (core, C#, and VB) implements the higher-level development experience expected from an IDE. This includes a number of concepts:

Immutable Project and Solution types, containing collections of source files and references, just like you work with in Visual Studio.
A Workspace class that contains a solution and manages changes (this is the mutable outer shell around the immutable projects & syntax trees).
Support for workflow features that are not actually part of the languages, such as loading MSBuild project files and parsing compiled XML doc comment files.
Implementations of basic IDE-level services such as source formatting, Find All References, rename, syntax highlighting, recommendations (the core parts of IntelliSense), code formatting, and a few other basic services for code editing.

Most importantly, these assemblies are still completely decoupled from Visual Studio, so you can easily use them to create your own IDE-like experiences.

This layer too is split into desktop and portable versions. All of the basic functionality remains in the portable assemblies; the desktop layers adds MSBuildWorkspace, which reads MSBuild csproj & vbproj files, and a couple of other desktop-specific APIs.

This layer, as well as all of the following layers, is wired together using MEF v2. MEF is used to provide implementations of Roslyn internal services (especially across layers), to provide services to the Visual Studio editor system, and to import user-provided extensions. For example, writing a custom codefix simply involves inheriting CodeFixProvider and exporting it via MEF using [ExportCodeFixProvider] (most user-provided services have custom [Export] attributes that provide additional metadata).

The one exception to this rule is custom diagnostic analyzers. The core compilation process, which invokes analyzers, does not use MEF; instead, the host passes a collection of AnalyzerReferences to the compiler (these come from Roslyn’s built-in analyzers, as well as any analyzer references in your project). Within Visual Studio, Roslyn’s VS layer has a (MEF-exported) VisualStudioWorkspaceDiagnosticAnalyzerProviderService which scans for VS extensions that contain analyzers (as a separate VSIX content type), allowing you to “export” analyzers in VS extensions as if it were a MEF export. However, since it isn’t actually MEF, you can’t [Import] things in these analyzers

Features

The Microsoft.CodeAnalysis.(CSharp

VisualBasic).Features assemblies implement advanced IDE features. These include the public base types to implement refactorings and diagnostics, as well as (as internal types) all of the refactorings & quick fixes in Visual Studio.

These assemblies are still decoupled from Visual Studio, so you can use them too in your own independent projects. (using MEF to import all of the internal implementations of the built-in services)

EditorFeatures

This layer actually connects all of the earlier layers to Visual Studio’s WPF editor system, implementing VS editor services and invoking all of the features from the previous layers.

This layer also implements other Visual Studio editor-based features that are tied to the language services, such as Metadata As Source, Peek Definition, Call Hierarchy, and more.

The only public APIs in this layer are a set of extension methods in Microsoft.CodeAnalysis.EditorFeatures.Text.dll that link Visual Studio APIs to Roslyn APIs, providing connections between VS ITextBuffers and Roslyn Documents. If you want to write normal VS extensions that consume Roslyn APIs (as opposed to Roslyn extensions like refactorings or diagnostics, which aren’t tied directly to Visual Studio at all), you will need this assembly to bridge the gap.

Unlike the previous layers, these assemblies aren’t standalone; they reference the (MEF-based) Visual Studio WPF editor APIs. However, they do not reference the rest of Visual Studio at all. In fact, you can set up the Visual Studio editor components in a MEF container and host the entire Roslyn editor (using this layer) outside of Visual Studio. For more details, see VSEmbed, where I did exactly that.

LanguageServices

Finally, Microsoft.VisualStudio.LanguageServices.dll (and the corresponding C# & VB DLLs) contains other, non-editor-related Visual Studio integrations, including the project system, debugger, object browser, options pages, and all of the other gory details of the various ways that Visual Studio interacts with the language services.

This layer is even less externally useful than EditorFeatures; it’s entirely coupled to ugly VS implementation details. Its only useful public type is the VisualStudioWorkspace class, which extends the core Roslyn Workspace with VS-specific integration methods to get projects, display definition & FAR results, and a few other things.

Next time: Inside the End-User Preview

Programming without errors – ErrorFree

2014-04-01T00:00:00+00:00

Errors are one of the most common and annoying problems in programming. Whether it’s the parser, the compiler, the type system, the runtime, or even system memory, everything you work with seems to have some way of complaining to you and refusing to run your code. Programmers spend more time fixing errors than any other task when developing applications ^{[citation needed]}.

But what if you could program without any errors?

To avoid these troublesome errors, I am proud to present a new language called ErrorFree. This language cannot have any errors, anywhere. In fact, every possible file, of any length, is a valid ErrorFree program that can compile and run successfully.

Design Philosophy

Avoiding every possible kind of error (while maintaining Turing-completeness) presents a number of challenges:

Syntax errors
To avoid all syntax errors, ErrorFree cannot have any kind of syntax. Syntactical constructions such as blocks, matching braces, or composite expressions all present opportunities for syntax errors, and are thus unacceptable.
Instead, ErrorFree is a stack-based language that is parsed one byte at a time. Every byte, except for pre-defined operator bytes, will simply push its numeric value onto the stack. Operator bytes will perform operations on the value(s) at the top of the stack.
For example, the ErrorFree program 12+ will push the values 49 and 50 on to the stack (remember that these are bytes, not ASCII or Unicode characters), then add them together, resulting in a stack containing the number 99.
Encoding errors
To avoid errors from invalid Unicode characters or UTF8-encoded sequences, ErrorFree source files are never treated as strings. Instead, each byte is interpreted directly. For programmer convenience, operator bytes are chosen based on their ASCII characters, making it easy to write ErrorFree code in a hex editor.
Stack underflow errors
Source files like +, which tries to operate on an empty stack, must also be valid. Therefore, the ErrorFree stack is pre-populated with an infinite number of zeros. Thus, it is never possible to run out of numbers on the stack.
Arithmetic errors
Since dividing by zero (including 0 ÷ 0) must also be valid, the ErrorFree stack is composed of double-precision floating-point numbers, so that these operations can simply return NaN or Infinity, instead of throwing an error. In contexts where an integer is required (eg, jumps or heap addresses), regular numbers are truncated, and NaNs and infinities are converted to zero.
Memory errors
Regrettably, most computers do not have unlimited storage capacity. If the stack grows too large to store, ErrorFree implementations are allowed to drop values from the bottom of the stack (above the cushion of infinite zeros) to reclaim space. Similarly, if the heap (see below) grows too large, the implementation is allowed to drop values from the farthest end from the address being inserted into.
Implementations are encouraged to avoid this behavior as much as possible.

Conventions

Every byte in an ErrorFree source file will either push a value to the stack (a value byte) or will perform an operation involving the stack (an operator byte). For convenience, operator bytes are chosen based on their ASCII values. Operator bytes fall into three categories:

Arithmetic operators use the appropriate symbols. For example, / will divide the top two values on the stack and push the result.
Pure/idempotent operators (operators that only depend on or affect the stack, and do not have side-effects or depend on external state) use lowercase letters symbolizing the operation. For example, t will transpose the top two values on the stack.
Impure operators (operators that have side-effects) use uppercase letters symbolizing the operation. For example, J will jump a number of bytes (the top of the stack) from the current position in the source file, then continue executing from that byte.

In addition to the stack, ErrorFree provides a heap, which can store numbers (double-precision floating point, just like the stack) at any positive or negative integral index. The heap can be used to store data, allowing you to build arrays, linked lists, or other more complex data structures. Like the stack, the heap is initialized to zero at every location.

ErrorFree does not provide a memory manager, so blocks of heap memory must be tracked by hand. Similarly, ErrorFree does not provide a call stack, so function calls and return pointers must also be tracked by hand in the heap.

Since raw bytes cannot be displayed as-is, ErrorFree source code should be displayed for reading as sequences of bytes in hexadecimal. For better readability, operator bytes should be displayed as their single ASCII representation, preceded by a space to maintain alignment. Newlines are meaningless, and can be used to group operations. For example:

01 02  +
04  * 05  *
0E  +

This code pushes the number 74 ((((1 + 2) * 4) * 5) + 0xE = 0x4A) on to the stack. (4A cannot be pushed directly, since J is an operator byte.) A simpler alternative would be 4B 01 -.

Operators

ErrorFree supports the following operators:

Arithmetic operators

All arithmetic operators will pop the values they read.

+
Adds the top two values on the stack and pushes the result to the stack.
-
Subtracts the top value on the stack from the second value on the stack, and pushes the result to the stack.
*
Multiplies the top two values on the stack and pushes the result to the stack.
/
Divides the second value on the stack by the top value on the stack, and pushes the result to the stack. 0/0 pushes NaN; dividing other numbers by zero pushes positive or negative infinity.
%
Pushes the second value on the stack modulo the top value.
^
Raises the second value on the stack to the power of the top value, and pushes the result.
=
Pushes one if the top two values on the stack are equal, or zero otherwise. NaN does not equal itself.
>
Pushes one if the second value on the stack is greater than the top value, or zero otherwise.
<
Pushes one if the second value on the stack is less than the top value, or zero otherwise.

Pure operators

d (Duplicate)
Pushes a second copy of the top value on the stack.
t (Transpose)
Transposes the top two values on the stack.
a (Absolute value)
Pops a value from the stack and pushes its absolute value.
s (Sign)
Pops a value from the stack and pushes its sign (-1, 0, or 1, or NaN).
r (square Root)
Pops a value from the stack and pushes its square root (or NaN if it’s negative).
l (Log)
Pops a value from the stack and pushes its base-10 logarithm (or NaN if it’s negative).
f (Floor)
Pops a value from the stack and pushes the largest integer smaller than that value. Leaves infinities and NaNs unchanged.
c (Ceiling)
Pops a value from the stack and pushes the smallest integer larger than that value. Leaves infinities and NaNs unchanged.

Impure Operators

C (print Character)
Pops a value from the stack, and prints its Unicode codepoint to standard out. Non-integer values are truncated, negative values are made positive, and infinity and NaN become 0.
N (print Number)
Pops a value from the stack and prints its numeric value to standard out.
D (the letter after C)
Reads a single character from standard in, and pushes its Unicode codepoint index on to the stack. Pushes -1 for EOF.
O (the letter after N)
Reads a number from standard in (followed by a newline) and pushes it onto the stack. The strings Infinity, -Infinity, and NaN are parsed as-is; all other letters or invalid characters are stripped. If there are no numeric characters to read (for example, if standard in is at EOF), this will push 0.
There is currently no way to determine whether a 0 was read because the user typed zero, because the user typed nothing but letters, or because standard in is at EOF.
R (Random)
Pushes a random number in the range [0, 1) on to the stack.
T (Timestamp)
Pushes the UNIX timestamp for the current time on to the stack.
J (Jump)
Pops a value from the stack, and moves the execution pointer that number of bytes from the current location in the file (forward or backward); execution continues from that byte. Non-integer numbers are truncated, NaNs and infinities are treated as zero, and indices wrap around after reaching either end of the file.
S (Save)
Pops two values from the stack, then stores the second popped value at the heap location of the first (topmost) value. Heap locations are truncated to integers; NaNs and infinities become zero.
L (Load)
Pops a value from the stack, then reads the heap location at that address and pushes its value to the stack.

Sample program

This 19-byte program prints the squares of numbers 1 through 0x42. It stores the loop index at heap location 0.

L 01  + 00  S
L  d  *  N
L 42  <
+  J
00

Here is a line-by-line explanation:

Increment the value at heap location 0.
(Load, add one, store)
Print the square of said value. (Load, duplicate, multiply, print)
Compare the value at heap location 0 to 0x42, pushing 1 if it’s less than 42 and 0 if it’s greater or equal. (Load, push comparand, compare)
Jump forward by two bytes if it was less (thus wrapping around to the beginning of the file), or one byte if it’s time to exit the loop. (add one, jump)
A final byte to jump to to exit the program. Without this byte, jumping forward would either wrap around (for 1) or remain at the jump instruction (for 0), so the program would hang after finishing the loop.

Other notes

It is impossible to make non-breaking changes to the ErrorFree language once it’s released, since any new code you want to allow already has an existing meaning. Instead, newer versions must be explicitly specified when invoking the compiler/interpreter.

ErrorFree does not include a built-in syntax for comments. Instead, you can simply write your comments directly in the code, then precede them by jump operators so that they don’t execute. When displaying code, comments can be written like operators. Care must be taken to ensure that no other jump instructions end up jumping into the middle of a comment. For example:

02  +
J  T  h  i  s     i  s     a     c  o  m  m  e  n  t
+

Farewell, MVP Program ... Hello, Google!

2014-03-30T00:00:00+00:00

After four exciting years, I am regretfully leaving the Microsoft MVP program. I have greatly enjoyed meeting and interacting with the Roslyn team, as well as the other C# & VB MVPs, and I hope to be able to keep up these connections at technical conferences. I’ve also savored the unique opportunity to discuss the future of C# and Visual Studio with the development teams at Microsoft, especially as new versions of these products are developed (I’ve filed over 150 bugs through the program).

In brighter news, I’m proud to announce that I’ve been hired by Google, working within the Google Docs team in the NYC office. Working for Google involves a wonderful corporate culture and a fantastic suite of internal tools, and I’m excited to contribute to Google’s products and improve the web for everyone.

Since Google and Microsoft are direct competitors, it is not possible to maintain both positions. Nevertheless, I hope to continue contributing to the C# community on StackOverflow, and to continue my open-source work on Visual Studio extensions (depending on how much spare time I have).

Keep coding!

Extending Visual Studio, part 5: Dealing with Unversioned Assemblies

2014-02-26T00:00:00+00:00

Last time, I talked about how to write Visual Studio extensions that work in multiple versions of Visual Studio, using the built-in assembly redirects.

Using unversioned assemblies is trickier. The most straightforward approach is to release a separate VSIX extension for each version of Visual Studio. This is the approach taken by Web Essentials, especially because it tends to be very tightly coupled to version-specific features within the web tools assemblies. However, this makes it much more annoying to maintain the separate codebases.

An alternative approach, taken by VsVim, is to put all code that uses unversioned assemblies in a separate project for each supported VS version. The primary extension assembly would then check the VS version at runtime to only load the wrapper assembly for the current version of Visual Studio. This approach is most convenient if you’re only using small bits of functionality from the unversioned assemblies, so that they can easily be abstracted out.

Finally, you can reference the assembly directly, then make sure that VS loads the correct version at runtime. This requires careful extra work to make sure that VS always ends up loading the correct assembly.

If you’re only using the unversioned assembly in XAML (eg, for themed controls from Microsoft.VisualStudio.Shell.ViewManager), you can simply reference the assemblies directly. As long as you don’t specify the version when declaring the xmlns: in your XAML file, the XAML will be compiled with a partial assembly name, and the runtime will bind it to the already-loaded assembly from the current VS version.

If you’re using it in code, this technique will not work. The C# compiler will always emit the full names of referenced assemblies, including the version number (taken from whatever version of the DLL the compiler loaded). Instead, you can redirect the assembly load at runtime with an AssemblyResolve handler, allowing your code to handle the request for the mis-versioned assembly and load the correct version instead.

The caveat in this approach is that you must add the AssemblyResolve handler before VS tries to load any type from the unversioned assembly. Otherwise, the CLR assembly loader will try to load the wrong version before you can catch it, and nothing will work. Even worse, because assembly loads are cached, once it fails to load the assembly once, it will never try again, even after you add your handler.

In particular, if your VSIX DLL exports MEF types (A MefComponent asset), the MEF catalog will call Assembly.GetTypes() to find all exports and imports, before any of your code has a chance to run. If any type in your assembly (including compiler-generated types for lambda expressions or iterators) has a field with a type from an unversioned DLL, the CLR will try to load that DLL immediately, and everything will break.

One way to solve this problem would be to write a module initializer that adds your handler as soon as your DLL is loaded, before the call to GetTypes(). Unfortunately, C# does not support module initializers, so unless you want to add a post-build step that uses ilasm & ildasm to inject an initializer by hand, this won’t help.

Instead, you simply need to make sure that you don’t have any types that directly reference unversioned assemblies. Make sure that you don’t have any fields (or base classes) of types from unversioned assemblies, and that you don’t use variables of those types in iterator methods or capture them in lambda expressions. If you violate this rule, your extension will fail with a TypeLoadException on any version of Visual Studio other than the version of the unversioned assembly that you bound to.

If you’re getting this exception, you can run the following code in LINQPad to figure out what types are causing problems:

const string BasePath = @"C:\Path\To\Solution";
const string ReferencesPath = BasePath + @"References\v10.0\";

// Load the assembly containing the types to test
var targetAssembly = Assembly.LoadFrom(BasePath + @"YourProject\bin\Debug\YourProject.dll");

// Load the versioned assemblies that VS will load successfully
Assembly.LoadFrom(ReferencesPath + "Microsoft.VisualStudio.Shell.10.0.dll");
Assembly.LoadFrom(ReferencesPath + "Microsoft.VisualStudio.Text.Data.dll");
Assembly.LoadFrom(ReferencesPath + "Microsoft.VisualStudio.Text.UI.Wpf.dll");

Type[] successfulTypes;
try {
	// Try to load the types without the unversioned assemblies
	targetAssembly
		.GetTypes()
		.Select(t => t.FullName)
		.Dump("Loaded types");
	Console.WriteLine("All types loaded successfully!");
	return;
} catch (ReflectionTypeLoadException ex) {
	successfulTypes = ex.Types;
	ex.LoaderExceptions.Select(e => new {
		Message = e.GetType().Name + ": " + e.Message,
		FusionLog = e is FileNotFoundException ? Util.OnDemand("Full Log", () => ((FileNotFoundException)e).FusionLog) : null
	}).Dump("Load Errors");
}

// Load the unversioned assemblies that will be handled by AssemblyResolve
Assembly.LoadFrom(ReferencesPath + "Microsoft.VisualStudio.CSharp.Services.Language.dll");

targetAssembly
	.GetTypes()
	.Except(successfulTypes)
	.Select(t => t.FullName)
	.Dump("Failed types");

Note that you will need to reset the query’s AppDomain (Ctrl+Shift+F5 in LINQPad) after you run it so that the unversioned assemblies won’t be loaded next time.

Next time: Tips from the trenches

Extending Visual Studio, part 4: Writing cross-version extensions

2014-02-25T00:00:00+00:00

Last time, I talked about the different approaches that Visual Studio takes toward assembly versioning.

Navigating this mess and producing a version-independent VSIX requires careful planning. The first step is to pick a minimum version of Visual Studio that you want to support. Your extension cannot use features or APIs that were introduced after this version. For most extensions, the minimum version would be VS2010, which introduced both the VSIX extension manager and the new WPF editor. However, if you want to extend newer features, such as Peek, the new CSS or HTMLX editors, or Roslyn, you will need to require a later minimum VS version.

Note that the format for VSIX manifests changed in VS2012. If you want to create an extension using VS2012 or later, and have it work on VS2010, you will need to manually rewrite the Source.extension.vsixmanifest file to use version 1 of the XML schema (this will break the designer). You can do that by copying the manifest from an existing 2010-compatible extension, such as mine, and editing all of the fields to match your extension. In particular, make sure to change the unique extension ID (this does not actually need to be a GUID; it can be convenient to place your extension’s name before the GUID to gain both uniqueness and readability).

Obviously, your extension cannot reference any immutable assemblies newer than your minimum VS target version. However, it is perfectly fine to add a reference to an immutable assembly installed with a newer version of VS; they are, after all, immutable.

When using versioned assemblies, you need to make sure that you don’t accidentally use APIs introduced in newer versions of VS. You also need to make sure that your extension is compiled only against the minimum versions of versioned assemblies; otherwise, it will still fail to load on older VS versions (the <bindingRedirect>s only redirect older versions, not newer ones).

To solve both of these problems, you need to make a local copy of the minimum versions of any versioned assemblies that you reference (typically in a folder named References under source control, and ideally in a subfolder by VS version for clarity). You can find these assemblies from any installed copy of the older VS versions (including Express Edition), or online from other open source projects that maintain this kind of compatiblity, such as VsVim, NuGet, or Rebracer. You must also add that directory to the ReferencePaths setting in the project to tell the compiler to find the DLLs from that folder; otherwise, the project won’t build if the older VS version isn’t installed. You can do this directly in the csproj file or in Project Properties in the Reference Paths section.

Next time: Using unversioned assemblies

Dissecting the new .Net Reference Source Browser

2014-02-24T18:30:00+00:00

The new .Net Reference Source Browser (see my previous post) is an excellent example of one of the less-obvious uses of the new Roslyn toolchain. Kirill Osenkov used Roslyn to recreate Visual Studio’s source browser experience in a standalone webapp, allowing people to browse the .Net Reference Source from anywhere. In this post, I will explore how this app is implemented.

The source browser is generated by a conversion tool that uses Roslyn to parse every source file in the codebase and generate a massive collection of HTML files comprising the full browser. Almost all of the work is done statically at build time; the only server-side code is the search engine. This makes it much faster to use, at the cost of consuming hundreds of thousands of files and a couple of gigabytes of disk space.

Generating output files

The converter tool loads every project file in the source release into a Roslyn workspace. It then traverses through every file in every assembly found in the workspace to generate source. As I mentioned last time, some of the projects reference assemblies that do not have source available (either because they’re written in managed C++ or for licensing reasons). For these projects, the converter uses Roslyn’s generated Metadata As Source files (from Visual Studio’s Go to Definition feature).

It runs each source file through the same Roslyn syntax highlighter used by VS itself, converting the result classification spans to HTML to generate the basic source code.

For the navigation features, each member is assigned a unique identifier, based on the standard doc comment ID (the identifiers used in compiled XML doc comment files). These identifiers are then hashed (using the first half of the MD5 to save space), and compiled into a giant index file (separate for each assembly) mapping the hash to the source file defining the member.

The definition of each symbol then gets this hash in its id="", so that adding the hash to the fragment in the URL to the source file will jump to the definition. Some identifiers are the definition for more than one symbol (eg, type declarations are also the definitions of their default constructors); for these, the identifier is wrapped in a <span> tag so that it has two separate IDs.

Javascript code in the index file looks up the hash from its URL fragment and redirects to the actual source file. Thus, you can navigate to a URL like https://referencesource.microsoft.com/mscorlib/a.html#266f59a804f72937 and end up at the actual definition of the symbol with that hash. This results in much shorter URLs than including the full path to the source file in the URL, and also allows other people to build URLs without knowing which file a symbol is defined (for example, this is how I built Ref12).

The converter runs each source file through the binding phase of the Roslyn compiler to get a semantic model mapping each source token to the symbol it refers to. This model is used to wrap each symbol in an HTML hyperlink pointing to its hash in the index file. When the link is clicked, Javascript code in the index file looks up the hash in the URL fragment, then redirects to the actual source file.

Similarly, it generates a static HTML file for each symbol containing a list of references to that symbol (each a hyperlink to a line of source). The converter wraps each member definition in a link to this file to power the Find All References feature.

For members defined in the same assembly, it generates links pointing directly to the source file defining the member; this saves the step of redirecting to the index file. For members defined in other assemblies, it links to a.html, so that assemblies aren’t tightly coupled to other projects’ source code.

Search

The search engine is the only piece of server-side code in the entire browser. It’s implemented using an ASP.Net Web API project, powered by the D.txt file for each assembly. Typing into the search box sends an AJAX request to https://referencesource.microsoft.com/api/symbols/?symbol=text, which finds all symbols matching the search query and replies with a set of HTML links to the definitions (through the a.html index mentioned above, so that the server doesn’t need to know where the results are defined in source).

Occasionally, you might see a search come back with Index is being rebuilt; this means that the API AppDomain was unloaded, so it needs to read the index data back into memory before it can perform any searches.

Icons

The icons used in the Reference Source Browser are the same icons used by Visual Studio itself. These are a numbered set of 236 different icons, and can be found at https://referencesource.microsoft.com/content/icons/0.png (through 235.png). The names of these icons come from the StandardGlyphGroup and StandardGlyphItem enums in Microsoft.VisualStudio.Language.Intellisense.dll. The first 190 icons come in groups of 6; one icon for every possible accessibility (including both FamOrAssem, which corresponds to C#’s internal keyword, and FamAndAssem, which is not supported by C#). You can run the following code (which requires a reference to Microsoft.VisualStudio.Language.Intellisense) to download all of the icons with the correct names:

string targetDir = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.MyPictures), @"Visual Studio Glyphs\");
Directory.CreateDirectory(targetDir);

foreach (StandardGlyphGroup g in Enum.GetValues(typeof(StandardGlyphGroup))) {
	if (g == StandardGlyphGroup.GlyphGroupUnknown) continue;

	if (!g.ToString().StartsWith("GlyphGroup")) {
		new WebClient().DownloadFile(
			"https://referencesource.microsoft.com/content/icons/" + (int)g + ".png",
			targetDir + (int)g + " - " + g.ToString().Replace("Glyph", "") + ".png");
		continue;
	}

	for (int i = 0; i < 6; i++) {
		int index = (int)g + i;
		new WebClient().DownloadFile(
			"https://referencesource.microsoft.com/content/icons/" + index + ".png",
			targetDir + index + " - " + g.ToString().Replace("GlyphGroup", "") + "-" + ((StandardGlyphItem)i).ToString().Replace("GlyphItem", "") + ".png");
	}
}

Exploring the new .Net Reference Source Code

2014-02-24T18:23:00+00:00

Seven years ago, Microsoft released the .Net Reference Source Code, containing source for much of the .Net BCL for use with debugging. You could configure Visual Studio to use Microsoft’s public symbols servers, then step into the actual source code of .Net methods and see what’s going on under the hood.

While this was, in theory, extremely useful, it had a number of limitations that made it far less useful. A number of the source files were blank, or simply missing entirely. The source code had been filtered by a poorly-made scrubber to remove all employee names, including those in actual source, resulting in lines like lock (s_[....]) or private void [....](). The debug symbols were rarely updated after .Net patches were released, so debugging wouldn’t work after installing updates to .Net. Most annoyingly, there was no way easy to browse the code independently to read the source for a particular type.

Today, thanks to the efforts of Kirill Osenkov, Alok Shriram, and others, Microsoft has redone the source release and solved all of these problems. The new version includes almost all of the actual source files, including things like attributes or enums that weren’t in the old releases. The scrubbing process has been fixed to only process comments & string literals so that all of the source files are syntactically correct. The release process has been simplified so that new symbols can be published for every .Net update.

Most importantly, you can now download a 50MB zip file containing all of the source files with their original structure, complete with csproj (and vbproj) files that you can open in VS. For the first time, you can open the .Net source code in VIsual Studio, and browse the source with the full VS experience, including Go To Definition, Find All References, quick search, and all of the other VS browsing features.

These are not the original csproj files from the actual BCL build, and nothing will actually compile. The compilation process for the BCL projects is far more complicated than a simple build, including auto-generated localization (SR) files, bits of native code, and various other specialized tools, including MSBuild magic for cyclic dependencies. However, they’re perfect for browsing within VS.

Reference Source Browser

However, the crown jewel of the new release is the new online source browser. Kirill Osenkov used Roslyn to create a wonderful web app that presents the full source code in an easily browsable fashion. You can search for any class or function name, then see complete syntax-highlighted source code right in your browser, without having to download or install anything (the search engine is actually more powerful than that; see the home page for more details). You can explore more directly by clicking any project from the home page to see all of the source files in that project. You can even see a full list of types in each project by namespace (click the namespace icon in any file), perfect for discovering little-known types in large assemblies.

Inside a source file, the true power of this tool is available. You can click any class or function name anywhere to immediately jump to its definition, allowing you to quickly discover exactly what obscure or internal methods do when trying to understand a complicated procedure (you can then hit Back to jump right back to where you were before). If you’re looking at a definition, you can click on it to see all references to that class or member anywhere in all of the reference source, allowing you to quickly see how complicated features are used, or to see where an internal API is surfaced. Because this is produced by the full Roslyn toolchain, this works with all references, including less-obvious ones like extension methods, type-inferred lambda expressions, etc.

As you navigate through the source, the URL will update to point to that exact file, so that you can always share the URL with other people to show what you’re looking at (this also works for Find All References). You can also click any line number to get a URL pointing that line (just like GitHub), perfect for showing people a particular piece of code.

Some parts of the .Net framework do not have source code in the browser, either because they’re written in C++/CLI (which Roslyn cannot parse), or because the relevant product teams have not yet completed the bureaucracy required to release the source publicly. To preserve Go to Definition within these projects, the source browser includes source from metadata for them (just like you get when from Go to Definition in Visual Studio, and in fact powered by the same Roslyn-based code). In the list of projects on the homepage, the ones without actual source simply have the DLL file as the filename, instead of the path to the .csproj file.

Visual Studio Integration

The one shortcoming of the new release is that you can’t jump directly to the source of a function from within Visual Studio. When debugging, you can step into functions that you call using the debug symbols, but you cannot quickly see the source as you write code in the first place.

To solve that, I created a Visual Studio extension called Ref12. After installing this extension on Visual Studio 2010 or later, you can press F12 on any function in the .Net class libraries and jump to the reference source for that function in your default browser (if you’re at Microsoft and actually working on the .Net framework, the extension won’t get in your way).

Easter Eggs

There are also a number of hidden URLs in the reference source containing additional stats:

https://referencesource.microsoft.com/i.txt contains stats for the overall index, including line count (over 6 million!) and more
https://referencesource.microsoft.com/mscorlib/i.txt (and for other projects) contains those same stats for individual projects
https://referencesource.microsoft.com/Projects.txt contains a list of project files (or DLL files from assemblies without source) included
https://referencesource.microsoft.com/Assemblies.txt contains a list of all assemblies in the source, along with the index for that project in Projects.txt, and the number of other projects that reference it
https://referencesource.microsoft.com/TopReferencedAssemblies.txt lists the number of other projects that reference each project, in descending order
https://referencesource.microsoft.com/System.Core/References.txt lists the assemblies that each project references
https://referencesource.microsoft.com/System.Core/ReferencingAssemblies.txt lists the assemblies that reference each project
https://referencesource.microsoft.com/System.Core/BaseMembers.txt lists, for each overriding class member, the containing assembly and unique hash of the base member that it overrides or implements
https://referencesource.microsoft.com/System.Core/d.txt lists every member in each assembly, including the short name, unique hash, member type, full name, and glyph index

Some of these files are useful for exploring the released source; most of the later ones are more useful for building new functionality on top of the source browser. For more information, see my next post, which will explain in detail exactly how the source browser works.

Next time: Under the hood

Extending Visual Studio, part 3: Assembly Versioning

2014-02-21T00:00:00+00:00

Last time, I talked about the core concepts and basic assemblies within Visual Studio that are used to create extensions. This time, I’ll talk about how Microsoft manages these assemblies across Visual Studio.

One of the complex aspects of writing Visual Studio extensions is dealing with versioning issues. Each release of Visual Studio changes the major version of all of the VS DLLs. Without some kind of work around, VS extensions would need a separate project for each version of Visual Studio, so that it can reference the correct versions of the VS DLLs.

Visual Studio takes a number of different approaches to this problem:

Immutable assemblies

Some VS assemblies simply never change, at all. These assemblies have exactly one version – the version of VS that they were introduced in – and newer versions of VS continue to ship the exact same DLL.

This is the simplest, most ideal solution to the problem; extensions can simply reference these DLLs directly, and they will continue to work with all future VS versions without any problems. However, since most VS functionality is designed to change in future versions, few parts of VS can live in immutable DLLs.

VS includes two categories of immutable DLLs: COM interop assemblies and immutable shell assemblies.

COM interop assemblies (Microsoft.VisualStudio.Shell.Interop.XX and similar) contain nothing but COM interfaces (including supporting enums and structures, but with no classes at all). Since COM does not allow existing interfaces to change, immutability isn’t a problem here. Instead, new functionality is added by creating new interfaces in newer COM assemblies (eg, IVsSolution through IVsSolution5) with additional methods.

Immutable shell assemblies (Microsoft.VisualStudio.Shell.Immutable.XX) contain managed types that the shell team has decided will never need to change. They mostly comtain simple types, such as interfaces, enums, and EventArgs classes, that are used by other parts of the shell.

Versioned assemblies

Then there are assemblies that do change, but which Microsoft officially supports for use in extensions. In order to make ordinary Visual Studio extensions work in future versions of Visual Studio, VS includes <bindingRedirect>s in devenv.exe.config that redirect all older versions of these assemblies to the correct version for the installed copy of VS. This way, extensions that reference older versions will be seamlessly redirected to the correct version on newer VS installations.

In turn, Microsoft must make sure to never introduce breaking changes in these assemblies. To facilitate this, versioned assemblies tend to mostly contain interfaces & officially-public APIs only; most of the implementations & internal APIs live in unversioned assemblies elsewhere (especially Microsoft.VisualStudio.Platform.VSEditor.dll)

Versioned assemblies include all of the public APIs for the new WPF editors (Microsoft.VisualStudio.Text.*), the core shell APIs (Microsoft.VisualStudio.Shell.XX), and a number of assemblies for specific features. See devenv.exe.config in your VS installation directory for a full list.

Unversioned assemblies

Finally, there are DLLs with no compatibility guarantees whatsoever. Most other VS assemblies, especially implementation assemblies and assemblies for smaller or younger teams, are unversioned assemblies, with a separate version for each VS release and no binding redirects at all.

There is no simple way to use unversioned assemblies in a VSIX without requiring a separate project for each VS release.

Next time: Writing cross-version extensions

Creating multiple Visual Studio “Profiles” with RootSuffixes

2014-01-27T00:00:00+00:00

When using Visual Studio, it can ocasionally be useful to have separate “profiles” – to be able to start multiple instances of Visual Studio with independent settings. For example, if you work on multiple projects that use very different source formatting settings, you can create a separate profile for each one, allowing you to open each project with the correct settings. If you create books or blog posts that have screenshots of Visual Studio, you can create a separate profile with pristine settings and no extensions installed, so that you can create screenshots that match the out-of-box experience.

Visual Studio supports this with the /RootSuffix switch. If you launch Visual Studio with /RootSuffix YourName, it will create new settings containers in the registry and in AppData with that name (HKCU\Software\Microsoft\VisualStudio\12.0YourName and <user profile>\AppData\Local\Microsoft\VisualStudio\12.0YourName). It will then ask you to sign in to a Microsoft account (for VS2013) and to pick a base settings file, just like launching Visual Studio for the first time.

To launch Visual Studio with a command-line parameter, type devenv /RootSuffix YourName into the Start menu / Start screen, then select the first result. You can also use the command prompt (from the VS installation directory), or copy the shortcut to VS from the start menu and add the parameter to the Target field.

This feature is usually used when testing Visual Studio extensions. VS extension projects will automatically register themselves in the /RootSuffix Exp hive (the “Experimental Instance”), which is launched when you hit F5. This allows you to test your extensions without interfering with normal development.

Generally, new RootSuffixes behave identically to, and completely independent of, regular Visual Studio instances. You can set all settings, maintain separate Recent lists, even install other extensions into the separate instance using Extension Manager.

However, there is no way to install an extension directly from a VSIX file into a separate RootSuffix. VSIX packages from the Visual Studio Gallery install as usual, but there is no way to install a VSIX from somewhere else (such as a private beta or a development version of an existing extension). Double-clicking a VSIX will install it into the primary instance, the VSIXInstaller command-line tool has no option to specify the RootSuffix, and you cannot install a VSIX by dragging it into Visual Studio.

To solve this, I wrote a command-line tool that installs VSIXs into any Visual Studio RootSuffix. You can download Root-VSIX from here, then install a VSIX to any VS RootSuffix on the command line:

Root-VSIX [<VS version>] <RootSuffix> <Path to VSIX>

The Perils of Uppercase in Gmail's Send As

2014-01-03T00:00:00+00:00

One of Gmail’s many useful features is “Send mail as”, which lets you use a single Gmail account to send emails from multiple email addresses. Especially when combined with email forwarding, this is a great way to manage multiple email accounts from a single Gmail tab.

By default, this feature will send the email from Gmail’s regular SMTP servers, including your actual email address in the Sender header to indicate the original sender of the email (without that, the email would be rejected as spam, since it isn’t coming from the correct SMTP servers for that domain). To avoid this, you can select Send through my SMTP servers to tell Gmail to connect to your domain name’s actual SMTP servers and send the email directly through them. This approach makes the email indistinguishable from regular emails sent from that domain name. To make it work, you need to enter the SMTP server, username, and password for Gmail to connect with.

However, when using this feature to send through a Google Apps domain with DKIM enabled, I noticed that every email I sent through Send as got an invalid DKIM signature, causing some email providers to reject it as spam (you can check this using port25’s verifier). Sending email directly over a regular SMTP client worked, but sending through Gmail’s Send As would break the DKIM signature.

After corresponding with Gmail support, I discovered that this happens if you use uppercase characters in the email address to send from. If you enter the address with only lowercase characters, it works fine; if you include uppercase characters, the signature breaks. If you’re having this problem, you need to delete the Send As entry, then re-create in all-lowercase.

Redirecting Assembly Loads at Runtime

2013-12-25T00:00:00+00:00

.Net’s assembly resolver (Fusion) takes assembly names (eg, MyCompany.MyProduct, Version=2.1.0.0, , Culture=neutral, PublicKeyToken=3bf5f017df1a30a5) and resolves them to actual DLL files, searching in the Global Assembly Cache, the folder containing the entry assembly, and any additional PrivateBinPaths registered for the AppDomain.

You can change the way it resolves specific assemblies using <bindingRedirect> tags in your App.config (or Web.config) file, giving it a different name to use instead if it tries to resolve a specific range of versions for an assembly. This is useful if you want to support references to multiple versions of the same assembly (eg, from older plugins), but only want to use a single version at runtime.

However, these redirects must be specified statically in the config file, and cannot be changed at runtime. If you don’t control the config file (eg, if you’re writing a plugin), or if you don’t know what redirects you will want until runtime, this doesn’t help.

Instead, you can handle the AppDomain.AssemblyResolve event, which lets you run your own code to load an assembly if the loader fails to find it.

This event is meant to be used if you put your assemblies somewhere where the loader can’t find them (eg, as embedded resources in your EXE for a single-file application, in a subfolder, or even over HTTP). However, you can also use this to catch loads of older versions of an assembly and load the newest version instead.

There are a few traps to beware of when writing AssemblyResolve handlers:

You must add your handler before the runtime tries to load the assembly. In particular, note that the JITter will load all assemblies used by a method before the method starts executing, in order to properly JIT-compile the method itself.
This means that you must add your AssemblyResolve handler before calling any method that uses types in the assembly in question.
Similarly, when you first reference a type, its static initializer will run, and the JITter will load any assemblies used by its static constructor (in addition to assemblies containing types in method signatures or fields).
Therefore, you must also add your handler before loading any type that uses the assembly directly.
The assemly resolver will only raise this event if it fails to find the assembly on its own. In particular, if you load an assembly that is in the GAC using its full name (including version and PublicKeyToken), the AssemblyResolve event will not be raised at all. This limits the utility of this event for binding redirection.
If you call Assembly.Load() in your handler with a more-precise assembly name, make sure to prevent stack overflows if that assembly name isn’t found either (the loader will raise the event again and re-enter your handler)
Once an assembly name fails to load, the runtime will cache the binding failure and never try to load that name again. In particular, if an assembly fails to load before you attach your handler, it’s too late for you to handle it next time.

I wrote code to handle AssemblyResolve and simulate a single binding redirect:

///<summary>Adds an AssemblyResolve handler to redirect all attempts to load a specific assembly name to the specified version.</summary>
public static void RedirectAssembly(string shortName, Version targetVersion, string publicKeyToken) {
	ResolveEventHandler handler = null;

	handler = (sender, args) => {
		// Use latest strong name & version when trying to load SDK assemblies
		var requestedAssembly = new AssemblyName(args.Name);
		if (requestedAssembly.Name != shortName)
			return null;

		Debug.WriteLine("Redirecting assembly load of " + args.Name
					  + ",\tloaded by " + (args.RequestingAssembly == null ? "(unknown)" : args.RequestingAssembly.FullName));

		requestedAssembly.Version = targetVersion;
		requestedAssembly.SetPublicKeyToken(new AssemblyName("x, PublicKeyToken=" + publicKeyToken).GetPublicKeyToken());
		requestedAssembly.CultureInfo = CultureInfo.InvariantCulture;

		AppDomain.CurrentDomain.AssemblyResolve -= handler;

		return Assembly.Load(requestedAssembly);
	};
	AppDomain.CurrentDomain.AssemblyResolve += handler;
}

Syntax-highlighted Markdown Code Blocks in Web Essentials

2013-12-10T00:00:00+00:00

After over two months of work, I rewrote the Markdown editor in Web Essentials to support syntax highlighting & IntelliSense for embedded code blocks.

If you add a GitHub-style fenced code block with a language identifier, Visual Studio will now provide full language services within that code block. You get the full editing experience you’re used to in Visual Studio, including syntax highlighting, IntelliSense, outlining, error checking, code snippets, and Peek Definition.

This is perfect for writing Readmes or documentation for open-source projects on GitHub, or for any other Markdown files you may write.

How it works

Visual Studio 2010 rewrote the editor from the ground up in WPF. This new editor has a powerful feature called Projection Buffers, which allow a single editor to seamlessly mix text from different sources. This can be used to make a virtual editor that mixes code from different files (eg, Debugger Canvas), or to mix different languages within a single file (eg, Javascript and CSS blocks within an HTML file).

My Markdown editor uses this feature to project code blocks in your Markdown files using Visual Studio’s built-in editor services

What languages are supported?

Any language that Visual Studio can highlight, including any custom extensions you’ve installed.

All languages services built for the new WPF-based editor should work perfectly with projection buffers. This includes HTML (sort of), CSS, Javascript, and any languages from extensions that only work in VS2010 or later. The language service itself needs to work properly with projection buffers; if it makes unwarranted assumptions about text buffers always having files on disk or only one TextView, things may not work.

If the language service needs to initialize things for its services to work (eg, a project system), I can still support it using an ICodeLanguageEmbedder, which Web Essentials will call to initialize whatever is necessary. If you have such a language service, feel free to send a pull request.

Before VS2010, an older system called contained languages allowed editors to be mixed like this. However, this system needed explicit integration from the language service being embedded. As far as I know, the only langauge services that support contained languages are C# & VB. I added an adapter layer to make these languages work as well (thanks to Jason Malinowski for spending lots of time getting this to work). If you have a different language that implements IVsContainedLanguageFactory, get in touch with me & I’ll add support.

To specify the language, use either the name of the ContentType or any extension (without the .) recognized by Visual Studio. If there are other names that should be recognized, send a pull request to add them to the map in ContentTypeExtensions.ContentTypeAliases.

Known Issues

Fenced code blocks don’t work in the Preview pane
We use MarkdownSharp to compile Markdown to HTML for the preview pane, and it does not support fenced code blocks. Until MarkdownSharp is upgraded (or until we find an alternative), we can’t support that.
Autoformat doesn’t work
The HTML auto-format engine does not work properly with embedded code blocks. This may be fixed in the next release of the web tooling.
Embedded XML, C++, and F# don’t get any syntax highlighting or IntelliSense
These languages are built using the old IVsColorizer infrastructure and cannot work in ProjectionBuffers. There is a separate service called IVsContainedLanguage that could allow old language services to work, but AFAIK, it is only implemented by C# & VB.
Highlighted ranges don’t update correctly for certain kinds of edits
Keeping track of where code blocks are as you edit the document (without spending too much time parsing) is complicated. If you notice it failing, file a bug and tell me exactly what you changed (please include the source file if you can).
Code blocks automatically indent too far
I’m not sure how fixable this is.
Background colors in syntax highlighting don’t appear
It looks like the background color for the Markdown code block overrides any background colors within the code itself. Other than disabling the background color for code blocks (in Options), I’m not sure if there is any way to fix this.

What's Next

So far, all I’ve done is provide the foundation for a modern Markdown editing experience. There are lots of other features that would make the Markdown editor more complete. If you would like to work on to any of these features, tweet me and I’ll give you advice.

Toolbar buttons & shortcut keys to toggle bold, italic, quotes, headers, lists
Pre-typing list or indented-code-block prefixes when pressing enter
IntelliSense, URL tooltips, & clickable hyperlinks for reference-style links (eg, [text][link name])
Smart tags to toggle between inline links and reference links
IntelliSense for URL paths in links
Separate IntelliSense & highlighting for YAML front-matter for Jekyll blog posts (#148)
Thumbnail tooltips for images
Separate config file with referenced assemblies & using statements for IntelliSense in C#/VB code blocks
Fix parser to code blocks in lists (which need 8 spaces)
Fix highlighting for quote blocks

Creating unused events in C#

2013-12-01T00:00:00+00:00

Some interfaces have events that most implementations will never raise. For example, the WPF ICommand interface has a CanExecuteChanged event that should be raised when the command becomes enabled or disabled. Most commands are always enabled, so this event is no needed. However, the interface still requires you to implement this event.
Thus, most ICommand implementations will look something like this:

class MyCommand : ICommand
{
	public bool CanExecute(object parameter) { return true; }
	public event EventHandler CanExecuteChanged;

	public void Execute(object parameter) {
		...
	}
}

This will generate a compiler warning, “The event ‘MyCommand.CanExecuteChanged’ is never used”.

To understand why events give a warning where unused methods don’t, you must understand more about how event work. .Net events are actually an accessor pattern, just like properties. For example:

public int MyProperty { get; set; }
public event EventHandler MyEvent;

The compiler transforms this code into something resembling

private int MyProperty_BackingField;
public int get_MyProperty() { return MyProperty_BackingField; }
public void set_MyProperty(int value) { MyProperty_BackingField = value; }


private EventHandler MyEvent;
public void add_MyEvent(EventHandler value) {
	// Thread-safe version of MyEvent += value;
}
public void remove_MyEvent(EventHandler value) {
	// Thread-safe version of MyEvent -= value;
}

Just like the compiler generates a backing field when you create an auto-implemented property, the compiler also generates a backing field when you create a field-like event. Unlike properties, event accessors only allow you to add or remove handlers from the event. To access the delegates currently in the event (eg, to call them and raise the event), you use the private backing field. This is why you can’t raise an event from outside the class that defines it – the backing field is private. To be more precise, the name MyEvent resolves to the backing field when used inside the class, and refers to the event (the accessor pair) when used elsewhere.

It is this auto-generated field the compiler is warning you about. Unlike an empty method (or a non-auto property with empty accessors), this backing field will needlessly waste memory for every instance of the class that you create. Therefore, just like any other field, the compiler will give you a warning if you never use the field.

To solve this warning, you need to get rid of the field. You can do that by explicitly defining event accessors that don’t do anything:

public event EventHandler MyEvent {
	add { }
	remove { }
}

Note that you should not make a field-like event virtual; the compiler does not handle overridden field-like events gracefully.

Extending Visual Studio 2013, Part 2: Core Concepts

2013-11-10T00:00:00+00:00

My previous post described how to get started writing Visual Studio extensions. This post will introduce the basic concepts needed to work with Visual Studio’s extensibility APIs.

Creating an extension

The Visual Studio SDK includes a number of templates for creating extensions. Which template you use depends on what parts of Visual Studio you want to extend.

Editor Extensions

If you only want to extend the new WPF editor, you can create an editor extension. Editor extensions allow you to add or change features (IntelliSense, highlighting, etc) for existing languages that use the new editor, as well as creating entirely new language service with their own syntax highlighting & IntelliSense. These projects simply export MEF components to be consumed by the editor, without involving any COM interop at all.

The SDK includes four different templates for editor extensions; these templates include sample code for specific editor features:

The Editor Classifier template contains a simple synax highlighting service for text files. You will need to write a parser to figure out which spans to highlight
The Editor Margin template creates a WPF control which will be displayed below the text editor. You can easily change where the margin will appear; you will need to write a useful UI to display in the margin.
The Editor Text Adornment template adds WPF visuals within the text, relative to individual characters. You will need to write code to determine where to place visuals, and design visuals & interactivity to display.
Finally, the Editor Viewport Adornment template adds UI to the text editor itself. You will need to figure out what UI to display, and what z-index it should get among the text editor’s existing layers.

Each of these templates includes a MEF export of the appropriate editor service (eg, IClassifierProvider, IWpfTextViewMarginProvider, or IVsTextViewCreationListener) for the editor to import and invoke. Extensions can export as many components as you want; you can freely mix and match exports from different templates in your project. If you want to extend a different part of the editor (eg, IntelliSense, outlining, or error checking), you can start from any of these templates, then delete the existing classes and start fresh.

VsPackage extensions

If you want to extend other aspects of Visual Studio, you’ll need to dig into the older COM code and create a VsPackage project. VsPackages can extend all other parts of Visual Studio, including tool windows, options pages, project system, and menu/toolbar commands. They can also add event handlers to other parts of Visual Studio to further customize behavior.

VsPackage extensions are created from the VsPackage project template, which will open a wizard allowing you to add sample functionality to your package. You can create a sample menu command, a custom tool window, or a fully custom editor (hosting your own WinForms control such as a designer rather than the built-in WPF text editor), then edit the samples to include your actual functionality.

More advanced extensions

Visual Studio extensions are packaged in VSIX files, which are ZIP files containing DLLs and other files used by the extension. When you install an extension, the VSIX installer will extract the VSIX to a folder for the extension, in <user profile>\AppData\Local\Microsoft\VisualStudio\12.0\Extensions\<random characters>. If your extension needs other files (eg, a separate EXE file), you can add the files to your project, then set Include in VSIX to true in the Properties window. You can get the installation directory in your code from Path.GetDirectoryName(typeof(YourType).Assembly.Location).

The VSIX file also controls how your DLL is loaded by Visual Studio; this is specified by the Assets section Source.extension.vsixmanifest file in the extension project. To load a VsPackage from your project, register it as a Microsoft.VisualStudio.VsPackage asset; to load MEF exports, use Microsoft.VisualStudio.MefComponent (the project templates do this for you). If you want to use both approaches in the same extension, you’ll need to register the project twice; once as each asset type.

Working with VsPackages

VsPackage extensions must contain a class that implements IVsPackage, which Visual Studio will call to intialize the extension. Most managed extensions will instead inherit the Package helper class, which implements most of this and other basic interfaces for you, leaving you free to write actual extensions. For more information, see MSDN.

In your package class, you can apply [Provide*] attributes to provide options pages, tool windows, project items, and other services. See here (scroll down to Provide) for all available attributes. You can also override the Initialize() method to add menu commands (using the IMenuCommandService service) and register event handlers.

The COM portions of Visual Studio are linked together using service providers. To get an instance of a VS-provided COM interface, you need to call GetService() on a ServiceProvider object and pass it the GUID for the service you need, from the GUID for the matching SVs* interface. It will return a COM object implementing the appropriate IVs* interface.

Your IVsPackage will get an IServiceProvider in its SetSite() method, which Visual Studio will call as the extension is initialized. The Package wrapper class implements this for you and saves a reference to the service provider, exposing a GetService() method for you to call:

var textManager = (IVsTextManager)this.GetService(typeof(SVsTextManager));

The COM interop classes for the various services you can use are contained in assemblies with names ending in .Interop; MSDN has a full list.

If you need to interact with newer MEF components from a VsPackage, you need to grab Visual Studio’s global composition service from from the ServiceProvider:

var componentModel = (IComponentModel)this.GetService(typeof(SComponentModel));

Working with MEF extensions

Using MEF is much simpler. You can export components used by the editor by implementing the appropriate editor interface, then decorating your class with the MEF [Exports] attribute. In these classes, you can import other editor services by decorating a field with the [Imports] attribute.

The new editor is built around ContentTypes, which map editor services to file types. Each editor language (C#, HTML, CSS, etc) has a ContentType containing the name of the language, as well as a list of base ContentTypes that it inherits. ContentType inheritance allows you to reuse editor service in more advanced languages; for example, the Razor ContentType inherits from the htmlx ContentType (HTMLX is the new HTML editor that shipped in VS2013; the old HTML editor is not easily extensible). You can create your own ContentTypes by applying attributes to MEF-exported static fields; see the documentation for details and a walkthrough.

Typically, you will only want to export services for certain ContentTypes, or for documents with certain roles (eg, read-only editors, editors that are backed by an actual file, as opposed to the output window, etc). You can control this by applying filtering attributes like [ContentType] or [TextViewRole] to your exported class; most editor components will filter by these attributes when importing.

For example:

[Export(typeof(IVsTextViewCreationListener))]
[ContentType("htmlx")]
class MyHtmlTextViewCreationListener : IVsTextViewCreationListener {
    [Import]
    public IVsEditorAdaptersFactoryService EditorAdaptersFactoryService { get; set; }

    [Import]
    public IClassifierAggregatorService Classifiers { get; set; }

    [Import]
    public ICompletionBroker CompletionBroker { get; set; }

    [Import]
    public IQuickInfoBroker QuickInfoBroker { get; set; }

    [Import]
    public SVsServiceProvider ServiceProvider { get; set; }

    public void VsTextViewCreated(IVsTextView textViewAdapter) {
        // ...
    }
}

For a full list of importable service, see MSDN. Note that this list only includes services from the editor itself; other parts of Visual Studio, such as the web tooling, export their own services that are not documented anywhere. To find these, use a decompiler.

If you need to use a COM-based service, you can get a ServiceProvider by [Import]ing an instance of the SVsServiceProvider interface, as shown in the code above.

Exportable editor services generally fall into two categories: listeners and providers. Listeners are used to handle events from the editor system. For example, you can export an IVsTextViewCreationListener and the editor will call your function whenever a text view is created. Similarly, IWpfTextViewConnectionListener will notify you when a TextBuffer is created for your ContentType, including new buffers in existing views (for projection buffers). Other VS components, such as the web tooling, have their own listeners.

Providers allow the editor to create instances of your services when creating text views of buffers. Since all MEF-managed parts are essentially singletons and cannot have multiple instances, the editor will import a single instance of an IWhateverProvider. This provider will expose a Create() method that takes a text buffer and returns an IWhatever for that document, allowing the editor to create as many instances as it needs.

For more information on the various services you can provide for the editor, see MSDN.

Next time: How Visual Studio deals with assembly versioning

Extending Visual Studio 2013, Part 1: Getting Started

2013-10-18T00:00:00+00:00

In addition to being an excellent development environment, Visual Studio also has a powerful extensibility system. In this blog post, I will explain how to start writing Visual Studio extensions, so you can make the IDE work the way you want it to.

Getting Started

To use or develop extensions, you need Visual Studio Professional or higher (Express Edition won’t work).

First, download and install the Visual Studio SDK (for VS2012, see here; this adds project types for Visual Studio extensions and is required in order to open or create any extension.

Next, you need to decide whether to add features to an existing open source extension or create a brand new extension.

Web Essentials
Web Essentials, by Mads Kristensen, is a collection of (mostly) web-related enhancements to Visual Studio. It includes lots of new IntelliSense completions (especially for CSS), new BrowserLink features, automatic JSHint for Javascript files, new warnings for HTML and CSS, and many other features. Any web-related functionality you want to add should probably go here.
SideWaffle
SideWaffle is collection of code snippets and templates (both single files and entire projects) for popular frameworks (Chrome extensions, Angular apps, Azure interactions, etc…). If you simply want to make a new template to create a preconfigured file or project, SideWaffle is the way to go. See this video for detailed instructions.
Your Name Here
If you want to create something that doesn’t fall into one of these categories, or if you’re creating something complex enough (or specialized enough) that it wouldn’t fit in Web Essentials, you can create your own extension, and upload it to the Visual Studio Gallery yourself.
To start from scratch, click File, New Project, select, Visual C# (or VB), Extensibility, then create a VSIX project. This will create an empty extension; you can also select one of the pre-built templates if they match your needs:

If you want to extend an existing extension, you’ll need to create an account on GitHub and fork the project to your account so that you can push your changes. Then, clone the project to your computer (GitHub for Windows is an easy way to do this; SourceTree is more powerful) and start making changes. Make sure to commit to every time you finish working on something so you have a nice history; you can do this directly from Visual Studio or in the aforementioned applications. Finally, once your feature is finished and tested, open a pull request on GitHub to ask the project maintainer to merge in your changes.

##Side Note: Tips for contributing with git The pull request model work best with feature branches – making a separate branch for each contribution. Before starting each feature, you’ll want to create a new branch, reset that branch to the latest version of the original (upstream) project, then make changes from there.

This is easiest to do on the git command line. First of all, if you aren’t already, use posh-git to get nice tab completion for git commands and branch names. (GitHub for Windows includes this if you use PowerShell; git bash has a similar feature)

First, you need to add the upstream repository as a remote (an external git repository that you can pull from) so that you can pull changes directly from the original project to your machine. This only needs to be done once:
git remote add upstream <url>
upstream is the name of the new remote; <git-url> is the https://github.com/... URL of the original repository. You can find this URL on the right side of the project on GitHub.com:

Once you’ve added the remote, run the following commands to make a new branch and reset to the original state.
Warning: This will blow away all uncommitted changes! Only do this after committing everything you’ve been working on.

git fetch upstream
git checkout -b my-awesome-feature
git branch reset upstream/master --hard

You’re now ready to begin working on the new branch. If the upstream project changes in the meantime, you can run the following commands to merge in the changes without breaking history: (make sure to commit everything first)
git fetch upstream
git rebase upstream/master
git rebase will rewrite the history of your local branch to include the upstream changes before your new commits. This way, when you make the pull request, all of your commits will extend the tip of the original repository, avoiding complex merge issues.

Since this rewrites history, if you’ve already pushed the branch to your fork on GitHub, your next push will need the -f flag to overwrite the existing remote history.

Visual Studio Extensibility Basics

Visual Studio is a 10+ year old codebase built on a mix of technologies new and ancient. Most of the older portions of Visual Studio, including the project system and the command dispatcher, are written in native code and support extensibility using ugly COM and GUIDs. Newer parts of Visual Studio are usually written in managed code (the big exception being the new Javascript language service, which is written in C++ around IE’s Chakra engine), and are thus easier to extend.

In particular, the editor was completely rewritten in Visual Studio 2010, and is implemented entirely in C# code using WPF. This new editor uses the Managed Extensibility Framework (MEF) to load and execute all of its internal components, making it very friendly for extensions. You can write classes that implement interfaces from the editor and export them using MEF, and the editor will automatically import and run them when it loads appropriate documents. Similarly, you can import interfaces from the editor to gain access to existing services like colorization and error checking for use within your extension.

Next time: Core concepts

Threads vs. Tasks

2013-10-11T00:00:00+00:00

.Net has three low-level mechanisms to run code in parallel: Thread, ThreadPool, and Task. These three mechanism serve different purposes.

Thread

Thread represents an actual OS-level thread, with its own stack and kernel resources. (technically, a CLR implementation could use fibers instead, but no existing CLR does this) Thread allows the highest degree of control; you can Abort() or Suspend() or Resume() a thread (though this is a very bad idea), you can observe its state, and you can set thread-level properties like the stack size, apartment state, or culture.

The problem with Thread is that OS threads are costly. Each thread you have consumes a non-trivial amount of memory for its stack, and adds additional CPU overhead as the processor context-switch between threads. Instead, it is better to have a small pool of threads execute your code as work becomes available.

There are times when there is no alternative Thread. If you need to specify the name (for debugging purposes) or the apartment state (to show a UI), you must create your own Thread (note that having multiple UI threads is generally a bad idea). Also, if you want to maintain an object that is owned by a single thread and can only be used by that thread, it is much easier to explicitly create a Thread instance for it so you can easily check whether code trying to use it is running on the correct thread.

ThreadPool

ThreadPool is a wrapper around a pool of threads maintained by the CLR. ThreadPool gives you no control at all; you can submit work to execute at some point, and you can control the size of the pool, but you can’t set anything else. You can’t even tell when the pool will start running the work you submit to it.

Using ThreadPool avoids the overhead of creating too many threads. However, if you submit too many long-running tasks to the threadpool, it can get full, and later work that you submit can end up waiting for the earlier long-running items to finish. In addition, the ThreadPool offers no way to find out when a work item has been completed (unlike Thread.Join()), nor a way to get the result. Therefore, ThreadPool is best used for short operations where the caller does not need the result.

Task

Finally, the Task class from the Task Parallel Library offers the best of both worlds. Like the ThreadPool, a task does not create its own OS thread. Instead, tasks are executed by a TaskScheduler; the default scheduler simply runs on the ThreadPool.

Unlike the ThreadPool, Task also allows you to find out when it finishes, and (via the generic Task<T>) to return a result. You can call ContinueWith() on an existing Task to make it run more code once the task finishes (if it’s already finished, it will run the callback immediately). If the task is generic, ContinueWith() will pass you the task’s result, allowing you to run more code that uses it.

You can also synchronously wait for a task to finish by calling Wait() (or, for a generic task, by getting the Result property). Like Thread.Join(), this will block the calling thread until the task finishes. Synchronously waiting for a task is usually bad idea; it prevents the calling thread from doing any other work, and can also lead to deadlocks if the task ends up waiting (even asynchronously) for the current thread.

Since tasks still run on the ThreadPool, they should not be used for long-running operations, since they can still fill up the thread pool and block new work. Instead, Task provides a LongRunning option, which will tell the TaskScheduler to spin up a new thread rather than running on the ThreadPool.

All newer high-level concurrency APIs, including the Parallel.For*() methods, PLINQ, C# 5 await, and modern async methods in the BCL, are all built on Task.

Conclusion

The bottom line is that Task is almost always the best option; it provides a much more powerful API and avoids wasting OS threads.

The only reasons to explicitly create your own Threads in modern code are setting per-thread options, or maintaining a persistent thread that needs to maintain its own identity.

LESS: Secrets of the Ampersand

2013-09-29T00:00:00+00:00

One of the less-documented features of the LESS language is the ampersand selector, which refers to the parent selector inside a nested selector.

The ampersand selector is most commonly used when applying a modifying class or pseudo-class to an existing selector:

a {
	color: blue;
	&:hover {
		color: green;
	}
}

The inner selector in this example compiles to a:hover. Without the &, it would compile to a :hover (a descendant selector that matches hovered elements inside of <a> tags), which is not what the author intended.

However, & has a variety of other uses.

Changing state based on parent classes

& can be used to make a nested selector that only applies its rules when an ancestor of the outer selector has a class. For example:

form a {
	color: purple;
	body.QuietMode & {
		color: black;
	}
}

This compiles to

form a {
  color: purple;
}
body.QuietMode form a {
  color: black;
}

This allows a QuietMode class to be applied to the root element to override loud styles, while keeping the quiet and “loud” (default) styles for each element in the same place.

This technique is particularly useful in mixins; it allows a mixin to to wrap a larger selector around the selector it was called in. For example:

.MyBox(@size, @color) {
	width: @size - 4px;
	border: 2px solid darken(@color, 10%);
	background: @color;

	body.SimpleMode & {
		width: @size - 2px;
		border: 1px solid @color;
		background: inherit;
	}
}

Using a prefix with the & allows the mixin to emit rules that are not nested in the location where the mixin was invoked, allowing it to specify higher-level filters.

Filtering a nested selector to only match certain elements

Similarly to the previous example, & can be used to create a nested selector (in a mixin or in another selector) that only matches a specific element. For example:

.quoted-source {
	background: #fcc;
	blockquote& {
		background: #fdc;
	}
}

This compiles to

.quoted-source {
	background: #fcc;
}
blockquote.quoted-source {
	background: #fdc;
}

This allows you to add speciallize a class for a specific element type, while keeping the specializations nicely nested within the rest of the class.

Avoiding repetition when selecting repeated elements

& can be used to repeatedly refer to the parent selector, while keeping your code DRY. For example:

.btn.btn-primary.btn-lg[disabled] {
	& + & + & {
		margin-left: 10px;
	}
}

This compiles to

.btn.btn-primary.btn-lg[disabled] + .btn.btn-primary.btn-lg[disabled] + .btn.btn-primary.btn-lg[disabled] {
	margin-left: 10px;
}

This example will add space between consecutive runs of large disabled buttons, without repeating the entire selector three times.

Simplifying combinatorial explosions

& can be used to easily generate every possible permutation of selectors for use with combinators. For example:

p, blockquote, ul, li {
	border-top: 1px solid gray;
	& + & {
		border-top: 0;
	}
}

This code will select all paragraph-like elements and add an upper border, then remove that border from such elements that already have one immediately preceding it.

The ampersands expand into all 16 possible combinations of these elements, without needing to type them by hand. These 6 simple lines expand into 24 lines or raw CSS:

p,
blockquote,
ul,
li {
  border-top: 1px solid gray;
}
p + p,
p + blockquote,
p + ul,
p + li,
blockquote + p,
blockquote + blockquote,
blockquote + ul,
blockquote + li,
ul + p,
ul + blockquote,
ul + ul,
ul + li,
li + p,
li + blockquote,
li + ul,
li + li {
  border-top: 0;
}

This technique works because LESS recognizes the comma combinator and expands nested selectors for each alternative.

Traditional OOP Inheritance in Javascript

2013-09-03T00:00:00+00:00

Javasript is not a traditionally object-oriented programming languague.

Wikipedia describes Javascript as a ”scripting, object-oriented (prototype-based), imperative, functional“ language. However, since most developers prefer to use classical (pun intended) object-oriented patterns, people have come up with ways to use Javascript with traditional OOP techniques, including classes and inheritance.

Most Javascript developers are by now familiar with the standard technique for Javascript classes. For example:

function Animal(name) {
	this.name = name;
	this.legs = [
		new Leg("LF"),
		new Leg("RF"),
		new Leg("LB"),
		new Leg("RB")
	];
}
Animal.prototype.summon = function() {
	ttsEngine.speak(this.name);
};
Animal.prototype.walk = function() {
	// Complicated code involving this.legs()
};

var myCow = new Animal("Betty");

The obvious way to add inheritance is to create a new function for the derived class, and set its prototype to an instance of the base class:

// This code is wrong!
function Animal() {	// Abstract
	this.legs = [];	// Child class must populate
}
Animal.prototype.walk = function() {
	// Complicated code involving this.legs()
};

function Kangaroo() {
	this.legs.push("L", "R");
	this.pouch = { };
}
Kangaroo.prototype = new Animal();

var skippy = new Kangaroo();

There are a number of problems with this code:

Kangaroo.prototype.constructor will be Animal, not Kangaroo.
Had the base Animal constructor taken parameters, this technique wouldn’t pass be able to those parameters on a per-instance basis
Every instance of the derived class will share the same array instance. After creating two more Kangaroo instances, they will appear to have 6 legs each.
More generall, since the base constructor is only executed once, any mutable state that it creates will be shared by every derived instance.

The first two problems are relatively easy to solve.

To solve the first problem, we can simply add

Kangaroo.prototype.constructor = Kangaroo;

However, this is not quite good enough. This will create an enumerable property, meaning that for in loops over instance of the derived class will include the constructor property. This behavior is contrary to the default constructor property, and is usually unexpected.

We can properly fix this by configuring the property using Object.defineProperty():

Object.defineProperty(
	Kangaroo.prototype, 
	"constructor", 
	{ 
		configurable: true,
		enumerable: false,
		writable: true,
		value: Kangaroo
	}
);

Note that Object.defineProperty() is not supported by IE 8 (or earlier); this problem cannot be fixed in that browser.

We can fix the second problem by explicitly calling the base constructor, making sure that it receives the correct this:

function Kangaroo(name) {
	Animal.call(this, name + " the Kangaroo");
	...
}

To fix the third problem, we need to make a new object inheriting Animal.prototype to be Kangaroo’s prototype, without actually calling the constructor. This is what the ES5 Object.create() function does:

Kangaroo.prototype = Object.create(Animal.prototype);

Since Object.create() takes an optional map of property descriptors, we can combine this with the second fix:

Kangaroo.prototype = Object.create(
	Animal.prototype,
	{
		"constructor": { 
			configurable: true,
			enumerable: false,
			writable: true,
			value: Kangaroo
		}
	}
);

This is now almost equivalent to the standard prototype created for each function, as detailed in the spec in section 13.2 (point 17).

We can move all of this code into a helper function:

function inherits(subConstructor, superConstructor) {
	subConstructor.prototype = Object.create(
		superConstructor.prototype,
		{
			"constructor": { 
				configurable: true,
				enumerable: false,
				writable: true,
				value: subConstructor
			}
		}
	);
}

We can then use it like this, making sure to call the superclass constructor inside the subclass constructor (that cannot be done automatically):

// This is the right way
function Animal(name) {	// Abstract
	this.name = name;
	this.legs = [];		// Child class must populate
}
Animal.prototype.walk = function() {
	// Complicated code involving this.legs()
};

function Octopus(name) {
	Animal.call(this, name + " the Octopus");

	this.legs.push(new Array(8).map(function(_, i) { return new Leg("L" + i); });
}
inherits(Octopus, Animal);

var q = new Octopus("Otto");

This function is exactly the Node.js util.inherits() function does, except that it also adds a super_ (static) property to the subclass constructor. For more details, see its source.

This still leaves one more difference from the standard prototype property; namely, the spec defines the prototype property as non-enumerable. We can fix that with another call to defineProperty:

function inherits(subConstructor, superConstructor) {
	var proto = Object.create(
		superConstructor.prototype,
		{
			"constructor": { 
				configurable: true,
				enumerable: false,
				writable: true,
				value: subConstructor
			}
		}
	);
	Object.defineProperty(subConstructor, "prototype", 	{ 
		configurable: true,
		enumerable: false,
		writable: true,
		value: proto
	});
}

This function creates a new prototype that fully complies to the spec.

Jekyll bug: Tag was never closed

2013-08-09T00:00:00+00:00

After upgrading to Jekyll 1.1, you may notice that posts that used to work fine now give an error like tag was never closed. These errors can appear for no apparent reason; the tags will appear to be correctly closed.

This error occurs if the first blank line in the post is inside a Jekyll block (eg, {% raw %} or {% highlight %}).

The bug is caused by a change in Jekyll’s post excerpt support.

As of version 1.0, Jekyll creates an excerpt for every post in your site, containing the first paragraph of text in that post (this allows post listings to display snippets of content). By default, the excerpt contains all text until the first blank line (double-newline); this separator can be changed by setting excerpt_separator in _config.yml.

The problem came in Jekyll 1.1.0, which parses Jekyll tags in post excerpts. Now, after extracting the first paragraph of text, the excerpt is parsed as Liquid markup so that tags in the excerpt work correctly. However, if you wrote posts without specifically designing them for excerpts, the excerpt can end in the middle of a Liquid tag, resulting in this error.

Here is an example of the bug:

---
title: "Buggy post"
---
This is an example of a post with some code:
{% highlight html %}
<html xmlns="http://www.w3.org/1999/xhtml">
<head><title></title></head>

<body></body>
</html>
{% endhighlight %}

Jekyll will extract the following excerpt:

This is an example of a post with some code:
{% highlight html %}
<html xmlns="http://www.w3.org/1999/xhtml">
<head><title></title></head>

This excerpt will result in Liquid Exception: highlight tag was never closed in buggy-post.txt, since the {% highlight %} tag is never closed.

To fix this, you can either fix every post so that the excerpt is valid Liquid markup, or prevent Jekyll from generating the excerpts in the first place.
Jekyll does not yet have a way to disable excerpts entirely, so the next-best option is to configure it so that the excerpts are always valid Liquid.

You can do this by setting excerpt_separator to a nonsense string that never appears in your posts, so that the excerpt will include the entire post (which is already known to be valid markup). Better yet, you can set excerpt_separator to an empty string, so that the excerpt will end immediately. This will reduce the amount of work that Jekyll needs to do, making your site build slightly faster.

In short, this bug can be fixed by adding the following line to _config.yml:

excerpt_separator: ""   # Workaround for https://blog.slaks.net/2013-08-09/jekyll-tag-was-never-closed/

Once #1386 is released, this line will disable excerpts entirely, adding another miniscule performance boost.

Immutability, part 4: Building lock-free data structures

2013-07-29T00:00:00+00:00

As I mentioned last time, the best way to create simple thread-safe lock-free data structures is with compare-and-swap loops.

To recap, this technique works as follows:

Fetch the current value of the field into a local variable

Run your actual logic to generate a new immutable object based on the current value (eg, push an item onto an immutable stack)

Use the atomic compare-and-swap operation to set the field to the new value if and only if no other thread has changed it since step 1.

If a different thread has changed the object (if the compare-and-swap failed), go back to step 1 and try again.

In C#, this algorithm looks like this:

/// <summary>Holds a reference to an immutable class and updates it atomically.</summary>
/// <typeparam name="T">An immutable class to reference.</typeparam>
class AtomicReference<T> where T : class {
	private volatile T value;
	
	public AtomicReference(T initialValue) {
		this.value = initialValue;
	}

	/// <summary>Gets the current value of this instance.</summary>
	public T Value { get { return value; } }
	
	/// <summary>Atomically updates the value of this instance.</summary>
	/// <param name="mutator">A pure function to compute a new value based on the current value of the instance.
	/// This function may be called more than once.</param>
	/// <returns>The previous value that was used to generate the resulting new value.</returns>
	public T Mutate(Func<T, T> mutator) {
		T baseVal = value;
		while(true) {
			T newVal = mutator(baseVal);
			#pragma warning disable 420	
			T currentVal = Interlocked.CompareExchange(ref value, newVal, baseVal);
			#pragma warning restore 420

			if (currentVal == baseVal)
				return baseVal;			// Success!
			else
				baseVal = currentVal;	// Try again
		}
	}
}

This class encapsulates a mutable reference to an inner type T, which must be immutable. The Mutate() function can be called to assign a new value based on the previous value, and will keep calling its callback until the current value does not change underneath it, as described above.

The #pragma warning directive is necessary to suppress a false positive about passing a volatile field as a ref parameter. Ordinarily, passing a field by reference will not preserve the field’s volatility, but the Interlocked methods are an exception to this rule. This is noted in the warning’s documentation.

Using this class with my earlier immutable stack, one can easily implement a thread-safe lock free mutable stack:

class AtomicStack<T> {
	private readonly AtomicReference<IStack<T>> stack 
		= new AtomicReference<IStack<T>>(PersistentStack<T>.Empty);
	
	public void Push(T item) {
		stack.Mutate(s => s.Push(item));
	}
	public T Pop(T item) {
		var oldStack = stack.Mutate(s => s.Pop());
		return oldStack.Peek();
	}
	public T Peek() {
		return stack.Value.Peek();
	}
}

The Push() and Pop() methods call their respective counterparts on the inner immutable stack, wrapped in the Mutate() operation so that they keep trying until the mutation succeeds.

Peek() simply gets the current value of the reference and peeks at the top of that stack. It is important to note that stack.Value can change at any time. Had this method needed to access the value twice, it would need to cache the current value of the field to prevent it from changing between two operations.
For example:

public IEnumerable<T> PeekTwice() {
	var current = stack.Value;
	yield return current.Peek();
	yield return current.Peek();
}

Compare-and-swap loops are the most efficient way to implement simple atomic lock-free data structures. However, writing thread-safe code is still intrinsically complicated; you still need to make sure to correctly use atomic operations and not operate on data that may change underneath you.

Immutability, part 3: Writing thread-safe data structures

2013-07-22T00:00:00+00:00

Last time, I showed how to create a simple covariant immutable stack.

One of the biggest attractions of immutable types is the ease with which they can be used to write thread-safe code. However, in the real world, things usually need to mutate.

How can this be done safely?

For an object to be safe, it must not be possible to observe it in an inconsistent state (as long as you follow its documented rules). For example, it should not be possible to see a collection with a hole in it from the middle of a resize.

For non-thread-safe objects, this is not very hard. As long as each function call leaves the object in a consistent state, it doesn’t take much effort to enforce this. It does mean that you must only raise events (or fire callbacks) when the object is in a consistent state. More subtly, you must only call virtual methods when in a consistent state, since a derived class can call arbitrary code outside the class. (this is why people criticize Java’s approach of making all methods virtual by default).
Since the object is documented (or assumed to be non-thread-safe, there is no need to ensure that other threads always see a consistent state.

For fully thread-safe objects, this requirement becomes much more difficult. Because other threads can interact with your object at any time, it must never be in an inconsistent state. There are a number of ways to make this work.

Atomic operations

Most instruction sets provide a couple of instructions which execute atomically (such as atomic increment or compare-and-swap). These instructions are inherently thread-safe; the CPU guarantees that no other thread can see inconsistent results. However, if your class is anything more complicated than a simple counter, this won’t help.

If you’re writing higher-level code, you can also use existing higher-level atomic operations, such as ConcurrentDictionary. These operations are in turn implemented using the other techniques described in this article.

Locks

This is the simplest approach. If you wrap every public method in a lock() statement, other threads can never observe inconsistent state, since they can only start executing after all method calls finish. This is the approach taken by Java’s Collections.synchronized*() methods.

However, this approach has a number of problems:

It’s slow. To guarantee safety, every method must prevent every other method from running, even for methods that would be safe to run concurrently, in case a third, unsafe, method is running too. This can be mitigated with ReaderWriterLocks, at the cost of additional complexity.
In addition, entering a lock is a kernel-level operation that comes with its own costs.
It’s prone to deadlocks if you lock on an externally visible object (such as this).
If thread A runs some other code that locks on the same object that you’re locking on, then starts waiting for some other lock, thread B can end up calling your function while holding that other lock. Thread B will then wait for thread A to finish before it runs your code, which won’t happen because thread A is stuck waiting for the lock that thread B is already holding.
It’s very prone to deadlocks if your code is re-entrant.
If thread A calls your code, then tries to grab some other lock in a re-entrant callback, but thread B is already holding that other lock and is now trying to enter your code, you will get a deadlock. This problem is difficult to fully prevent when writing reusable re-entrant code.

Compare-and-swap loops

To avoid locks, you must only use primitive atomic operations. This is where immutable data structures really shine. If you put all of the state you need to change in a single immutable object, you can make all of the changes from an ‘offline’ reference to the object, then atomically swap in your new copy if no-one else has changed it. This technique works as follows:

Fetch the current value of the field into a local variable
Run your actual logic to generate a new immutable object based on the current value (eg, push an item onto an immutable stack)
Use the atomic compare-and-swap operation to set the field to the new value if and only if no other thread has changed it since step 1.
If a different thread has changed the object (if the compare-and-swap failed), go back to step 1 and try again.

It is important that the object being swapped be fully immutable. Otherwise, changes to parts of the object from other threads can go unnoticed during the atomic replacement, leading to the ABA problem.

Using deeply immutable objects prevents this from causing a problem. If A is truly immutable, there can be nothing wrong with missing the B that happened in the middle; if there was any change that wasn’t fully undone, the new value wouldn’t be the same A.

This technique only works if the mutation you’re trying to perform (step 2) is a pure function. Since the operation can run more than once, any side effects that it has can happen multiple times, which will probably break your program. Similarly, it must be thread-safe (which pure functions implicitly are), since nothing is preventing it from running on multiple threads concurrently.

If the function does have side-effects, you can still use this technique by reverting those side effects if the compare-and-swap failed (in step 4). This way, the side effects from each invocation will be cancelled before the next try. However, this can only help if the function and its inverse (to revert changes) are both thread-safe, and if you can guarantee that the brief window of time before the side-effects of a failed attempt are reverted won’t cause trouble. If not, the only solution is a normal lock.

Make every state “consistent”

Compare-and-swap loops are the ideal way to implement lock-free mutable data structures. However, compare-and-swap can only operate on one thing at a time. If you need to mutate two separate fields, you need more complicated techniques.

If you’re mutating two independent fields, compare-and-swap can still be used. You can simply put all of the state you’re modifying into a single immutable class, then perform all of the mutations together inside a single compare-and-swap loop.

If you’re mutating two fields that are connected to each-other, this isn’t possible. A lock-free queue is a good example of this. A queue needs a head field that points to the first node (for dequeue), and a tail field that points to the last node (for enqueue).

The dequeue operation can be built using an ordinary compare-and-swap loop just like a stack; create a new node that points to the current head and swap out the old head for it.

However, enqueuing an item has two discrete steps. First, you need to set the next pointer of the current tail node to point to the new node, so that it can be dequeued when the head reaches the current tail. Then, you need to set the tail field to the new node, so that the next enqueue operation starts from the new node.

These two operations cannot be done in the same compare-and-swap loop, since both fields need to be set atomically. Otherwise, another thread can start inserting a second new element before tail is updated, and one of the nodes will be swallowed.

The obvious way to fix this would be to use a lock to prevent other threads from enqueuing an item between these two steps. However, you can also do this in a lock-free fashion. You simply need to make sure that if another thread runs between the two operations, it will first finish the job that the previous thread started. In other words, before inserting any new node, check whether there is a half-inserted node from another thread and finish inserting it first.

This technique looks roughly like this

Use a compare-and-swap loop to set the tail field to tail.next, to finish a half-done insertion from a different thread.
Use a compare-and-swap loop to set the tail.next to the new node (only if no other thread has set it to a different node at the same time)
Use a compare-and-swap loop to set tail to the newly-inserted node, but only if no other thread did so in the meantime from step 1.

Steps 1 and 2 must be done in the same loop in case a third thread inserts a second item after this thread finishes an earlier pending insertion.

The dequeue operation then becomes much more complicated, because it needs to work correctly in all three states. For more information, see Julian Bucknall’s detailed implementation of a lock-free concurrent queue.

The basic idea is to design the public methods to be able to run successfully from any intermediate state caused by another thread. This technique can be very difficult to implement correctly for complex operations.

As with simple compare-and-swap loops, the individual mutations must be either pure or safely revertible.

Final notes

The idea behind all of these techniques is to either handle or prevent other threads that modify your object while your operation is running. When using thread-safe objects, you must bear the same requirement in mind. Any time you perform two operations on an object, it is your responsibility to ensure that nothing has changed between the two operations.

To aid in this, well-designed concurrent classes will offer composite operations that perform common tasks atomically, such as .Net’s ConcurrentDictionary.GetOrAdd() method) (this is one of the problems with Java’s Collections.synchronized*() wrappers). In the absence of such methods, you will still need to use locks to ensure that no other mutations interrupt your operation.

Next time: How to build thread-safe lock-free data structures using compare-and-swap loops

Immutability, part 2.5: Adding covariance to the immutable stack

2013-06-28T00:00:00+00:00

Last time, I showed how to create a simple immutable stack. However, this stack is not covariant. Ideally, we should be able (for example) to implicitly convert IStack<string> to IStack<object>.

Most collection types cannot be co-variant. Had List<string> been convertible to List<object>, you would then be able to add an int (or any other type) to the converted List<object>, even though it can’t fit in List<string> (which the casted instance actually is). To be precise, covariance is only type-safe for immutable types.

Since our Stack<T> class is immutable, we should be able to simply change it to public interface Stack<out T> and get co-variance instantly.

In practice, it’s not so simple. Even though the class is immutable, it still has a method that takes T as a parameter (namely, Push(T)). Even though the method doesn’t mutate anything, it still wouldn’t be type safe:

IStack<string> stringStack = PersistentStack<string>.Empty;
IStack<object> objectStack = stringStack;
objectStack = objectStack.Push(4);
Console.WriteLine(objectStack.GetType());	// This is PersistentStack<string>

This isn’t safe because covariance is a property of the interface, not the implementing class. Even when casted to IStack<object>, the instance is still an IStack<string>, and therefore its Push(T) method can still only take a string. Similarly, the return value of Push() would still have been IStack<string>, which would be implicitly converted to IStack<object> to fit in the variable.

The actual behavior that we want for Push() is a little more subtle. If we have an IStack<string>, we should be able to push any object onto it, resulting in a new stack whose type is the least common ancestor between the type of the original stack and the type that we pushed. For example, Stack<Button>.Push(new TextBox()) should return a Stack<Control>.

Therefore, the desired signature for Push() would look like this:

public interface IStack<out T> {
	IStack<U> Push<U>(U element) where T : U;
	// Other members...
}

In other words, Push() can return a stack of any supertype of our element type. Unfortunately, this won’t work; C# (unlike Java) does not support upper bounds on a generic type parameter. You can write X<T> where T : SomeType, but not X<T> where SomeType : T.
Java, by contrast, would allow us to write Stack<U> <U super T> push(U element);.

However, all is not lost. Although we can’t constrain Push() based on the parent type’s type parameter, we can constrain it on its own type parameter. Specifically, we can make it an extension method:

public static IStack<U> Push<T, U>(this IStack<T> stack, U element) where T : U;

In fact, once we move it out of the class itself, we don’t even need a second type parameter. Since the stack is covariant, the IStack<T> parameter is convertible to IStack<U>, so we can simplify it to take just one generic parameter:

public static IStack<U> Push<U>(this IStack<U> stack, U element);

This way, you can write stringStack.Push(new object()), and the compiler will infer U to be object, then covariantly convert the IStack<string> to IStack<object> to pass as the first parameter.

Note that C#’s type inference algorithm will not attempt to find the least-common-ancestor when presented with two types (unless one is a descendant of the other), so you would need to specify the type parameter explicitly: buttonStack.Push<Control>(new TextBox()). Rather, it can only infer ancestral relationships can be inferred, such as buttonStack.Push(new Control()).

The only remaining problem is that the extension method must be specific to one implementation. It can extend (and return) the IStack interface, but it must create a specific implementing type. There is no elegant type-safe way for it to return whatever implementing type was passed to it.

Here is the code for a fully covariant immutable stack:

public interface IStack<out T> {
	IStack<T> Pop();
	T Peek();
	bool IsEmpty { get; }
}

public static class PersistentStackExtensions {
	public static IStack<TNew> Push<TNew>(this IStack<TNew> stack, TNew element)  {
		return new PersistentStack<TNew>.LinkNode(stack, element);
	}
}

public abstract class PersistentStack<T> : IStack<T> {
	public static readonly PersistentStack<T> Empty = new EmptyNode();

	private class EmptyNode : PersistentStack<T> {
		public override IStack<T> Pop() {
			throw new InvalidOperationException("Stack is empty");
		}
		public override T Peek() { 
			throw new InvalidOperationException("Stack is empty");
		}
		public override bool IsEmpty { get { return true; } }
	}

	internal class LinkNode : PersistentStack<T> {
		readonly IStack<T> previous;
		readonly T element;
		
		public LinkNode(IStack<T> previous, T element) {
			this.previous = previous;
			this.element = element;
		}

		public override IStack<T> Pop() {
			return previous;
		}
		public override T Peek() { return element; }
		public override bool IsEmpty { get { return false; } }
	}
	public abstract IStack<T> Pop();
	public abstract T Peek();
	public abstract bool IsEmpty { get; }
}

IStack<string> stringStack = PersistentStack<string>.Empty;
stringStack = stringStack.Push("A").Push("B");

IStack<object> objectStack = stringStack.Push<object>(42);
	
while (!objectStack.IsEmpty) {
	Console.WriteLine(objectStack.Peek());
	objectStack = objectStack.Pop();
}

The private PersistentStack<T>.LinkNode class now needs to be internal so that the extension method can instantiate it.

Next time: Swapping immutable objects without losing thread-safety

Immutability, part 2: Creating a simple immutable stack

2013-06-23T00:00:00+00:00

Last time, I explained the basic meaning of immutability. The simplest useful example of an immutable class is an immutable stack. Immutable stacks work just like regular stacks – with Push(), Pop(), and Peek() methods – except that instead of mutating the original instance, Push() and Pop() return a new, modified, instance.

In code, that looks like

public interface IStack<T> {
	IStack<T> Push(T element);
	IStack<T> Pop();
	T Peek();
	bool IsEmpty { get; }
}

IStack<int> myStack = empty;
myStack = myStack.Push(1).Push(2).Push(3);
	
while (!myStack.IsEmpty) {
	Console.WriteLine(myStack.Peek());
	myStack = myStack.Pop();
}

Each implementation of this interface would supply a singleton empty instance; since they are immutable; there is no need to have more than one empty instance. Thus, there is no need for a constructor. Since Pop() needs to return the new stack, it cannot return the removed item; therefore, Peek() is the only way to get the item.

As a side benefit, this pattern naturally enables and encourages fluent interfaces, since each mutation methods returns a new instance. (eg, myStack.Push(1).Push(2).Push(3))

The naive implementation of this interface would use a list, and copy the list when creating a new stack:

public class NaiveStack<T> : IStack<T> {
	private readonly ReadOnlyCollection<T> items;
	private NaiveStack(IList<T> items) {
		this.items = new ReadOnlyCollection<T>(items);
	}
	
	public static readonly NaiveStack<T> Empty 
		= new NaiveStack<T>(new T[0]);
	
	public IStack<T> Push(T element) {
		var list = new List<T>(items);
		list.Add(element);
		return new NaiveStack<T>(list);
	}

	public IStack<T> Pop() {
		if (IsEmpty)
			throw new InvalidOperationException("Stack is empty");
		var list = new List<T>(items);
		list.RemoveAt(list.Count - 1);
		return new NaiveStack<T>(list);
	}

	public T Peek() { return items.Last(); }
	public bool IsEmpty { 
		get { return items.Count == 0; } 
	}
}

The problem with this approach is that each mutation requires an O(n) copy to create the list behind the new instance. This makes Push() and Pop() awfully slow, and vastly increases the memory footprint until the intermediate instances can be GCd.

To solve these problems, we can design the stack as a persistent data structure. Instead of copying anything when pushing on to a stack, we cab make the new stack instance hold only the newly added item, and maintain a reference to the previous instance storing the rest of the items. Popping from the stack can then simply return the previous instance. Basically, the stack becomes an immutable single-linked list.

public abstract class PersistentStack<T> : IStack<T> {
	public static readonly PersistentStack<T> Empty = new EmptyNode();

	public IStack<T> Push(T element) {
		return new LinkNode(this, element);
	}

	private class EmptyNode : PersistentStack<T> {
		public override IStack<T> Pop() {
			throw new InvalidOperationException("Stack is empty");
		}
		public override T Peek() { 
			throw new InvalidOperationException("Stack is empty");
		}
		public override bool IsEmpty { get { return true; } }
	}

	private class LinkNode : PersistentStack<T> {
		readonly IStack<T> previous;
		readonly T element;
		
		public LinkNode(IStack<T> previous, T element) {
			this.previous = previous;
			this.element = element;
		}

		public override IStack<T> Pop() {
			return previous;
		}
		public override T Peek() { return element; }
		public override bool IsEmpty { get { return false; } }
	}
	public abstract IStack<T> Pop();
	public abstract T Peek();
	public abstract bool IsEmpty { get; }
}

In this implementation, the empty and non-empty nodes are different enough that I decided to use polymorphism rather than if statements. The PersistentStack<T> type contains the only common logic (Push()); everything else is implement by the two concrete classes.

Here, the empty stack truly is a singleton; only one instance will ever be created for each element type. It can only be used as a starting point to create new stacks; the other methods simply throw exceptions. It also serves as the final node in the chain for all non-empty stacks.

Pushing onto any stack (empty or not) returns a new node that holds the new item and points to the previous stack; popping a non-empty stack simply returns that reference.

Like other linked lists, this approach implements both Push() and Pop() in O(1) in both memory and time.

Next time: Adding covariance

Immutability, part 1: Read-only vs. Immutable

2013-06-11T00:00:00+00:00

A read-only object is an object that does not expose any way to change it. ReadOnlyCollection<T> (returned by AsReadOnly()) is a good example). However, IEnumerable<T> is also read-only.
The important distinction is that read-only objects are allowed to change.

If you write list.Add(new Test()), your read-only collection (which just wraps list) will have changed.

Read-only collections are useful for designing safe APIs, where only the owner of the collection is allowed to change it.
However, it won’t do any good for thread-safety.

An immutable object is an object that cannot change at all, no matter what happens (Reflection doesn’t count). string is an excellent example of an immutable class; the value of an existing string instance can never change, no matter what happens. (barring reflection, unsafe code, and certain marshalling tricks)

.Net does not have any built-in immutable collections, but the CLR team released a library of immutable collection types on NuGet.

Truly immutable objects are intrinsically thread-safe. Since there is no way to modify them, there is no chance that other threads will observe an inconsistent instance.

In summary, there are four kinds of “unchangeable” things:

readonly (ReadOnly in VB.Net; final in Java) fields.
These fields cannot be re-assigned to point to a different instance. However, nothing prevents you from mutating the instance pointed to by the field.
Read-only classes
These classes do not expose any APIs to mutate their contents, but can change by other means (typically, by changes made directly to the mutable objects they wrap, or in response to events handled within the class).
Examples include ReadOnlyCollection<T>.
API-enforced immutable classes
These classes are not intrinsically immutable, but have an API design that guarantees that mutations will not happen. In other words, they may contain mutable state, but they will never actually mutate that state. Examples include delegates (MulticastDelegate has an array of handlers, which it will never mutate).
Compiler-enforced immutable classes
These classes only contain readonly fields that are themselves immutable. This means that the compiler will report an error if the developer mistakenly tries to mutate it. This provides an additional layer of defence against potential mistakes.
Examples include most EventArgs classes.

Finally, an object is deeply immutable (or read-only) if both it and all of its properties are also deeply immutable (or read-only). In other words, the entire graph of objects reachable from it must be immutable (or read-only).

Next time: Creating a simple immutable stack

Writing the endraw tag in Jekyll code blocks

2013-06-10T00:00:00+00:00

Last time, we saw how to write about Jekyll tags in Jekyll-based blog posts, using HTML entities or the {% raw %} block.

These techniques cannot be used in syntax-highlighted code blocks (Jekyll’s {% higlight %} tag or a Markdown code block), since such blocks are always HTML-escaped. Instead, you can wrap all of the code in the block with a Liquid {% raw %} tag. Since the Liquid tag is processed before Markdown or syntax highlighting, this works perfectly.

{% raw %}
Liquid uses tags like {% if %} or {% for %}.
It also supports variable interpolation: {{ someVariable }}
{% endraw %}

This approach works wonderfully, until you try to put the {% endraw %} tag in a code block. You can’t put that in a {% raw %} block, because it will close the block. You can’t use entities, because text within highlighted code blocks is always HTML-escaped.

One potential option would be to break apart the tag; something like

{% raw %}
This is how you show the termination of the `{% raw %}` tag inside itself: 
{% endraw %}

However, due to a bug in Liquid, this doesn’t work correctly. This markup is supposed to mean the following:

Write the literal text {% within the {% raw %} block
Terminate the {% raw %} block with {% endraw %}
Write the rest of the of the tag (endraw %})
Re-enter the {% raw %} block to continue with other markup

What actually happens is that Liquid ignores the actual {% endraw %} command, and treats the entire line as raw text. In other words, the code you see in the above example is (incorrectly) rendered exactly as written. The literal text {% before the {% endraw %} causes Liquid to ignore the tag completely, breaking this technique.

To work around this bug, we need to put the {% text outside the {% raw %} block. However, Liquid does not have any way to escape a {, and we can’t use HTML escaping inside the highlighted code block, so there is no obvious way to write the {% without breaking the parser.

Variables can help here. We can create a Liquid variable that holds the literal text {%, then interpolate this variable outside the {% raw %} block. For example:

{% assign openTag = '{%' %}
{% raw %}
This is how you show the termination of the `{% raw %}` tag inside itself: 
{% endraw %}{{ openTag }} endraw %}{% raw %}
This content is back inside the {% raw %} block
{% endraw %}

This code is parsed like this:

Terminate the {% raw %} block with {% endraw %}
Write the value of the openTag variable ({%) with {{ openTag }}
Write the remainder of the {% endraw %} tag as literal text with endraw %}
Re-enter the {% raw %} block for additional raw content

This technique can also be used to write tags in inline code: `{{ openTag }} sometag %}`

If these workarounds seem complicated, writing this post itself (also in Liquid Markdown!) was even more complicated.
See the source to see how I did it.

Writing about Jekyll in Jekyll

2013-06-09T00:00:00+00:00

Jekyll is a very nice system for writing blogs. However, it does have some shortcomings, particularly in the Liquid templating engine. In this post, I will talk about how to write about Liquid tags within a Liquid file (such as a Jekyll blog post). The problem is that writing Liquid syntax such as tags or variables in the content will cause Liquid to interpret them as commands, and break evertything.

This problem can occur when writing a blog post about Liquid itself (such as this one), or when writing a blog post about code that generates Liquid markup (like I did earlier).

You may want to mention Liquid syntax in two ways: inline code (mentioning a {% tag %} in a sentence), and multi-line syntax highlighted code blocks (including a whole chunk of Liquid markup in a post).

Liquid includes a tag specifically designed to solve this problem: the {% raw %} tag. For example, the Liquid Markdown used to produce the preceding sentence is:

Liquid includes a tag specifically designed to solve this problem: 
the `{% raw %}{% raw %}{% endraw %}` tag.

I wrap the text I’m trying to produce ({% raw %}) in {% raw %}...{% endraw %} tags to prevent liquid from treating the content itself as a tag. Since Liquid processes the text before the Markdown parser sees it, the resulting Markdown source is simply `{% raw %}`, which is exactly what I want.

All that overhead is very annoying when writing a simple tag. We can make it simpler by writing part of the tag as an HTML entity, so that Liquid doesn’t recognize it as a tag: {% tag %}. The browser will display the HTML entity as a regular {, but Liquid won’t recognize it. However, this does mean that we can’t use Markdown ` blocks to create the code block, since that will escape the HTML and make it display a literal {.
Thus, the final approach is <code>{% tag %}</code>. Although this isn’t much shorter than the Liquid raw tag, I find it more readable, since it doesn’t nest Liquid content within Liquid commands.

Next time: How can you do this inside a syntax-highlighted code block?

Migrating client-side syntax highlighting to Jekyll

2013-06-02T00:00:00+00:00

The next step in migrating my blog to Jekyll was to convert the code blocks to use Jekyll’s {% highlight %} tag.

Since Blogger has no support for server-side syntax highlighting, all of the code blocks in my HTML are implemented as <pre class="brush: someLanguage">...</pre>, with HTML-escaped code inside the tag. (the class name is used by SyntaxHighlighter) I needed to convert that to Liquid tags with raw (non-escaped) code inside of them.

To do this, I wrote a small C# script:

const string PostsFolder = @".../_posts";
var langMappings = new Dictionary<string, string>{
	{ "vb", "vb.net" }
};
Func<string, string> GetLanguage = lang => {
	string mapped;
	if (langMappings.TryGetValue(lang, out mapped))
		return mapped;
	return lang;
};

Func<string, MatchEvaluator> Replace = replacement => ManifestResourceInfo => ManifestResourceInfo.Result(replacement);

var regexes = new [] {
	Tuple.Create<Regex, MatchEvaluator>(
		new Regex(@"\s*\<pre\s*class=""brush:\s*(\w+);?\s*"">\n*(.*?)\n*</pre>\s*", RegexOptions.Singleline),
		m => "\n{% endraw %}\n{% highlight " + GetLanguage(m.Groups[1].Value) +" %}\n" 
			+ WebUtility.HtmlDecode(m.Groups[2].Value) 
			+ "\n{% endhighlight %}\n{% raw %}\n"
	),
	Tuple.Create(new Regex(@"{% raw %}\s*{% endraw %}\s*", RegexOptions.Singleline), Replace("")),
	Tuple.Create(new Regex(@"<font face=""Courier New"">([^<]+)</font>", RegexOptions.Singleline), Replace("<code>$1</code>"))
};

foreach (var file in Directory.EnumerateFiles(PostsFolder, "*.html")) {
	File.WriteAllText(file,
		regexes.Aggregate (File.ReadAllText(file), (txt, tuple) => tuple.Item1.Replace(txt, tuple.Item2))
	);
}

(This kind of script is most easily run in LINQPad or scriptcs)

This code loops through every HTML file in PostsFolder and runs the text through a series of regular expressions (yes, evil) to update it for Jekyll.

The first, and biggest, regex matches <pre class="brush: someLanguage">...</pre>, and convers each to Jekyll {% highlight someLanguage %} blocks. Since SyntaxHighlighter uses different language names than Pygments’, I have a langMappings that maps language names from the HTML to language names for the Jekyll output. I also un-HTML-escape the contents of each code block. Finally, because I modified blogger2jekyll to wrap each post in a {% raw %} tag, it terminates the {% raw %} and re-enters it after the code.

Depending on what your code blocks look like, you may want to change it to wrap the contents of each {% highligh %} tag in a {% raw %} block too.

The next regex strips empty {% raw %} blocks in case there are two code blocks in a row (which I had here).

The last regex replaces bad non-semantic <font> tags created by Windows Live Writer with <code> tags.
If your imported HTML has similar issues, you can add more regexes to correct them.

Next time: Preserving the RSS feed

Migrating from Blogger to Jekyll

2013-05-31T00:00:00+00:00

The first step in my migration to Jekyll was to import my old posts into the Jekyll site. To do this, I used blogger2jekyll, a wonderful open-source Node.js script that does exactly that.

Using this tool is very simple. First, log into Blogger’s admin panel, got to Settings, Other, and click Export blog to download a giant XML file with all of your posts.

Next, install and run the script: (you’ll need to install Node.js first)

npm install -g blogger2jekyll
blogger2jekyll  /path/to/blog-dd-mm-yyyy.xml ./_posts

If you aren’t running it from the directory containing your Jekyll site, you’ll need to specify the full path to Jekyll’s _posts directory.

This script will create HTML files with the contents of each post from the exported blog, ready for Jekyll to serve. It will include layout: "post" in the Jekyll front matter; if you have a different layout name, you’ll need to do a bulk replace within the resulting files. It will also set the permalink for each post so that existing posts keep their old URLs (even if the Jekyll blog hasa different URL scheme).

I had a couple of problems with this script:

When running on Windows, the generated permalink URLs use backslashes rather than forward slashes
Posts that contain Liquid-like markup will break Jekyll with a parse error
The script also imports comments directly into the resulting HTML, which is rarely a good idea.

I fixed all of these issues in a pull request, which has been merged and pushed to npm.

I changed it to wrap the contents of each post in a Liquid {% raw %} tag. I also added an internal option to skip comments. However, I didn’t add a command-line interface for the comment option; to use it, you’ll need to manually add , skipComments: true to the parse() call.

Next time: Converting Code Blocks

About the new design

2013-05-27T00:00:00+00:00

My new design is powered by Jekyll and LESS (the LESS does as much or more as the Jekyll).

When implementing the design, I had the following goals in mind:

No costs
I use GitHub Pages for completely free hosting (other than the cost of the domain name)
This means that I cannot use Octopress or Jekyll plugins
No build step
I want to be able to edit posts from anywhere, without having to install Ruby or Grunt.js and run any kind of build process before pushing
I do use pre-compiled LESS, since Jekyll on GitHub Pages cannot compile LESS, and I really want to use LESS. (also, since LESS needs to be edited far less often than post content, and since my editor automatically compiles LESS files on save)
No Javascript
Especially with the power of CSS selectors, there should be no reason to use Javascript for static content
If I add comments, I will have to relax this restriction (since the site will no longer be purely static content)
DRY implementation
Repetition of data or rules is a problem not only in large programs, but also in HTML templating engines or style definitions. LESS in particular is very helpful in getting rid of unnecessary repetition in CSS.
Rich category support
I feel that category listing pages are a great way to make posts more discoverable
Clean, minimalist, design
One of the mistakes of my old design was that it put too little focus on content (both in size and in coloring).
Easy navigation within series of multiple posts
Coming soon
For now, the new sidebar in post pages (see left side) provides this to a limited extent
Excessive use of bulleted lists
Bulleted lists are an excellent way to concisely present structured data

Next time: How I migrated from Blogger to Jekyll

Relaunch!

2013-05-26T00:00:00+00:00

After nearly a year of inactivity, I have finally returned to my blog.

I had neglected it for so long primarily because I don’t like Blogger’s compose options. After spending so much time on StackOverflow and GitHub, I find Markdown far more convenient than Windows Live Writer or Blogger’s compose window, especially when writing about code. I also was never very happy with the design I originally created, and doing raw HTML / CSS design in Blogger is painful.

To solve these problems, I just finished porting my blog to Jekyll on GitHub Pages. Now that I can write posts in my favorite Markdown editor, I hope to end up writing much more.

In the next few posts, I’ll write about how I designed the new blog and how I migrated my existing content.

My old blog is still up at old-blog.slaks.net. However, all of the content has been migrated to the new one (with the same URLs), so that should not be necessary. If you see anything wrong with the new site, let me know on Twitter or by email.

Visual Studio 2012 and webpages:Version

2012-06-11T22:33:00+00:00

If you open an older ASP.Net MVC3 project in Visual Studio 2012, you may see lots of errors in the Razor views, along the lines of “The name 'model' does not exist in the current context”, and similar errors whenever you try to use MVC features like HTML helpers or ViewContext (eg, “System.Web.WebPages.Html.HtmlHelper does not contain a definition for TextBoxFor”).

This happens if there is no <add key="webpages:Version" value="1.0" /> in the <appSettings> element in Web.config.

Without this element, Visual Studio will assume that you’re using the latest version of Razor and the WebPages framework. Until VS2012, this wasn’t a problem, since there was only one version. However, since VS2012 ships with ASP.Net WebPages 2.0, the IDE will load this version by default. Since you’ve specified the MVC integration & <configSection> for 1.0 (in Views/Web.config, since all of the assembly references specify Version=1.0.0.0), the language services won’t load the MVC settings. Therefore, the @model directive will not work, and the view will inherit the standard WebPage base class rather than the MVC WebViewPage base class (which contains the MVC HTML helpers)

This issue has no effect at runtime because the server won’t load any version 2.0 assemblies.

Exploring Caller Info Attributes

2012-03-01T06:57:00+00:00

Last year, Microsoft announced a simple new feature in C# 5: Caller Info Attributes. These attributes let you to create methods with optional parameters and tell the compiler to pass the caller’s filepath, line number, or member name instead of the parameter’s default value. This allows you to create logging methods that automatically know where they’re being called.

When the feature was announced, I wrote a couple of blog posts that delved into some of the corner cases of the new feature. At the time, there was no public implementation, so they were pure conjecture.

This morning, Microsoft released the beta of Visual Studio 11, which is the first public build supporting these attributes. Now, I can finally test my theories. Here are the results:

Although these classes are new to the .Net Framework 4.5, you can still use this feature against older framework versions by creating your own classes in the System.Runtime.CompilerServices namespace. However, the feature will only work if the code calling the method is compiled with the C# 5 compiler; older compilers will ignore the attributes and simply pass the parameters’ default values.

All of the attributes can only be applied to arguments of types that have standard (not custom) implicit conversions to int or string. This means that it isn’t practical to overflow [CallerLineNumber] (the compiler ran out of memory first), so I can’t test how that behaves.

Using [CallerMemberName] on field initializers passes the field name, and on static or instances constructors passes the string ".cctor" or ".ctor" (as documented) In indexers, it passes "Item".

If a class has a constructor that takes only caller info attribute parameters, and you create another class that inherits it and does not declare a constructor (thus implicitly passing optional parameters), it passes the line number and file name of the class keyword in the derived class, but leaves the declared default for the member name (I suspect that’s a bug).

If you do declare a constructor, it passes the string ".ctor" as the member name for the implicit base() call (just like a normal method call from inside a constructor) and the line number of the beginning of the constructor declaration. If you actually write a base() call, it passes the line number of the base keyword.

If a call spans multiple lines, [CallerLineNumber] passes the line containing the openning parenthesis.

Delegates are fully supported; if you call a delegate that has an argumented annotated with a caller info attribute, the compiler will insert the correct value, regardless of the method you’re actually calling (which the compiler doesn’t even know).

LINQ query comprehension syntax is not supported at all; if you create a (for example) Select() method that contains a caller info attribute, then call it from a LINQ query (not lambda syntax), the compiler will crash (!). (they will fix that)

Expression trees do not support optional parameters at all, so that corner case is irrelevant.

Attributes are the most interesting story. What should happen if you declare a custom attribute that takes parameters with caller info attributes, then apply that attribute in various cases? This could potentially be very useful, since there is currently no way for an attribute to know what it’s being applied to. (I hadn’t thought of this usage when I wrote the original blog post)

The documentation says that this will work in all cases, and that [CallerMemberName] will pass whatever the attribute is being applied to. However, in the beta build, this doesn’t always work.

Attributes applied to method arguments or return values do not pass any caller info at all. Attributes applied to types or generic type arguments do not pass member names (this is very disappointing)

Hopefully, those will be fixed before release.

ASP.Net MVC Unobtrusive Validation Bug

2012-02-21T18:15:00+00:00

If you use the ASP.Net MVC 3 [Compare] validation attribute on a model property, then include that model as a property in a parent model (so that the field name becomes Parent.ChildProperty), the built-in unobtrusive client validation will choke, and will always report the field as having an error.

This is due to a bug on line 288 of jquery.validate.unobtrusive.js:

adapters.add("equalto", ["other"], function (options) {
    var prefix = getModelPrefix(options.element.name),
        other = options.params.other,
        fullOtherName = appendModelPrefix(other, prefix),
        element = $(options.form).find(":input[name=" + fullOtherName + "]")[0];	// Bug

    setValidationValues(options, "equalTo", element);
});

Because the value of the name attribute selector is not quoted, this fails if the name contains a ..

The simplest fix is to add quotes around the concatenated value. However, the jQuery selector there is overkill. HTML form elements have properties for each named input, so you can do this instead:

adapters.add("equalto", ["other"], function (options) {
    var prefix = getModelPrefix(options.element.name),
        other = options.params.other,
        fullOtherName = appendModelPrefix(other, prefix),
        element = options.form[fullOtherName];
    if (!element)
        throw new Error(fullOtherName + " not found");
    //If there are multiple inputs with that name, get the first one
    if (element.length && element[0])        
        element = element[0];
    setValidationValues(options, "equalTo", element);
});

Protecting against CSRF attacks in ASP.Net MVC

2012-01-25T23:18:00+00:00

CSRF attacks are one of the many security issues that web developers must defend against. Fortunately, ASP.Net MVC makes it easy to defend against CSRF attacks. Simply slap on [ValidateAntiForgeryToken] to every POST action and include @Html.AntiForgeryToken() in every form, and your forms will be secure against CSRF.

However, it is easy to forget to apply [ValidateAntiForgeryToken] to every action. To prevent such mistakes, you can create a unit test that loops through all of your controller actions and makes sure that every [HttpPost] action also has [ValidateAntiForgeryToken].

Since there may be some POST actions that should not be protected against CSRF, you’ll probably also want a marker attribute to tell the test to ignore some actions.

This can be implemented like this:

First, define the marker attribute in the MVC web project. This attribute can be applied to a single action, or to a controller to allow every action in the controller.

///<summary>Indicates that an action or controller deliberately 
/// allows CSRF attacks.</summary>
///<remarks>All [HttpPost] actions must have 
/// [ValidateAntiForgeryToken]; any deliberately unprotected 
/// actions must be marked with this attribute.
/// This rule is enforced by a unit test.</remarks>
[AttributeUsage(AttributeTargets.Class | AttributeTargets.Method)]
public sealed class AllowCsrfAttacksAttribute : Attribute { }

Then, add the following unit test:

[TestMethod]
public void CheckForCsrfProtection() {
    var controllers = typeof(MvcApplication).Assembly.GetTypes().Where(typeof(IController).IsAssignableFrom);
    foreach (var type in controllers.Where(t => !t.IsDefined(typeof(AllowCsrfAttacksAttribute), true))) {
        var postActions = type.GetMethods()
                                .Where(m => !m.ContainsGenericParameters)
                                .Where(m => !m.IsDefined(typeof(ChildActionOnlyAttribute), true))
                                .Where(m => !m.IsDefined(typeof(NonActionAttribute), true))
                                .Where(m => !m.GetParameters().Any(p => p.IsOut || p.ParameterType.IsByRef))
                                .Where(m => m.IsDefined(typeof(HttpPostAttribute), true));

        foreach (var action in postActions) {
            //CSRF XOR AntiForgery
            Assert.IsTrue(action.IsDefined(typeof(AllowCsrfAttacksAttribute), true) != action.IsDefined(typeof(ValidateAntiForgeryTokenAttribute), true),
                            action.Name + " is [HttpPost] but not [ValidateAntiForgeryToken]");
        }
    }
}

typeof(MvcApplication) must be any type in the assembly that contains your controllers. If your controllers are defined in multiple assemblies, you’ll need to include those assemblies too.

The Dark Side of Covariance

2011-12-15T04:14:00+00:00

What’s wrong with the following code?

var names = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
...
if (names.Contains(sqlCommand.ExecuteScalar())

This code is intended to check whether the result of a SQL query is contained in a case-insensitive collection of names. However, if you run this code, the resulting check will be case-sensitive. Why?

As you may have guessed from the title, this is caused by covariance. In fact, this code will not compile at all against .Net 3.5.

The problem is that ExecuteScalar() returns object, not string. Therefore, it doesn’t call HashSet<string>.Contains(string), which is what it’s intending to call (and which uses the HashSet’s comparer). Instead, on .Net 4.0, this calls the Enumerable.Contains<object>(IEnumerable<object>, string) extension method, using the covariant conversion from IEnumerable<string> to IEnumerable<object>. Covariance allows us to pass object to the Contains method of any strongly-typed collection (of reference types).

Still, why is it case-sensitive? As Jon Skeet points out, the LINQ Contains() method is supposed to call any built-in Contains() method from ICollection<T>, so it should still use the HashSet’s case-insensitive Contains().

The reason is that although HashSet<String> implements ICollection<string>, it does not implement ICollection<object>. Since we’re calling Enumerable.Contains<object>, it checks whether the sequence implements ICollection<object>, which it doesn’t. (ICollection<T> is not covariant, since it allows write access)

Fortunately, there’s a simple fix: just cast the return value back to string (and add a comment explaining the problem). This allows the compiler to call HashSet<string>.Contains(string), as was originally intended.

//Call HashSet<string>.Contains(string), not the
//covariant Enumerable.Contains(IEnumerable<object>, object)
//https://blog.slaks.net/2011/12/dark-side-of-covariance.html
if (names.Contains((string)sqlCommand.ExecuteScalar())

(I discovered this issue in my StringListConstraint for ASP.Net MVC)

CAPTCHAs do not mitigate XSS worms

2011-12-08T02:32:00+00:00

One common misconception about web security is that protecting important actions with CAPTCHAs can prevent XSS attacks from doing real damage. By preventing malicious code from scripting critical tasks, the idea goes, XSS injections won’t be able to accomplish much.

This idea is dangerously wrong.

First of all, this should not even be considered except as a defense-in-depth mechanism. Regardless of whether the actions you care about are protected by CAPTCHAs, XSS attacks can create arbitrary UI on your pages, and can thus make “perfect” phishing attacks.

Also, even with CAPTCHAs, an XSS injection can wait until the user performs the critical action, then change the submitted data to the attacker’s whim.

For example, if Twitter took this approach to prevent XSS injections from sending spammy tweets, the attacker could simply wait until the user sends a real tweet, then silently append advertising to the tweet as the user submits it and fills out the CAPTCHA.

However, there is also a more fundamental issue. Since the injected Javascript is running in the user’s browser, it simply display the CAPTCHA to the user and block all page functionality until the user solves the CAPTCHA. The attacker can even put his own text around the CAPTCHA to look like a legitimate security precaution, so that the (typical) user will not realize that the site has been compromised. (that could be prevented by integrating a description of the action being performed into the CAPTCHA itself in a way that the attacker can’t hide)

I haven’t even mentioned the inconvenience of forcing all legitimate, uncompromised users to fill out CAPTCHAs every time they do anything significant.

In summary, CAPTCHAs should only be used to prevent programs from automatically performing actions (eg, bulk-registering Google accounts), and as a rate-limiter if a user sends too many requests too quickly (eg, getting a password wrong too many times in a row).

XSS can only be stopped by properly encoding all user-generated content that gets concatenated into markup (whether HTML, Javascript, or CSS)

About Concurrent Collections

2011-11-20T18:42:00+00:00

One of the most useful additions to the .Net 4.0 base class library is the System.Collections.Concurrent namespace, which contains an all-new set of lock-free thread.

However, these collections are noticeably different from their classical counterparts. There is no simple ConcurrentList<T> that you can drop into your code so that it will become thread-safe. Instead, the new namespace has a queue, a stack, and some new thing called a bag, as well as ConcurrentDictionary<TKey, TValue> that largely resembles classical dictionaries. It also has a BlockingCollection<T> class that wraps a concurrent collection and blocks until operations can succeed.

Many people have complained that Microsoft chose to provide an entirely new set of classes, rather than adding synchronized versions of the existing collections.

In truth, however, creating this new set of classes is the correct – and, in fact, only – choice. Ordinary synchronized are rarely useful and will not make a program thread-safe. In general, slapping locks everywhere does not make a program thread-safe! Rather, that will either not help (if there aren’t enough locks) or, if there are enough locks, result in deadlocks or a program that never runs more than one thread at a time. (depending on how many different objects get locked)

Collections in particular have two fundamental issues when used on multiple threads.

The first, and simpler, issue is that the collection classes themselves are not thread-safe. If two threads add to a List<T> at the same exact time, one thread is likely to overwrite the other thread’s value. A synchronized version of List<T> with a lock around every method would solve this problem.

The bigger issue is that any code that uses a List<T> is unlikely to be thread-safe, even if the list itself is thread-safe. For example. you can never enumerate over a multi-threaded list, because another thread may change the list at any time, invalidating the enumerator. This issue could be solved by taking a read lock for the lifetime of the enumerator. However, that is also a bad idea, since if any client code forgets to dispose the enumerator, the collection will deadlock when written to.

You also cannot use indices. It is never safe to get, set, or remove an item at an index, because another thread might remove that item or clear the entire collection between your index check and the operation.

To solve all of these problems, you need thread-safe collections that provide atomic operations. This is why all of the concurrent collections have such strange methods, including TryPop, AddOrUpdate, and TryTake. These methods perform their operations atomically, and return false if the collection was empty (as appropriate; consult the documentation for actual detail). Thus, the new concurrent collections can be used reliably in actual multi-threaded code without a separate layer of locks.

Beware of Response.RedirectToRoute in MVC 3.0

2011-11-09T04:00:00+00:00

ASP.Net MVC uses the new (to ASP.Net 3.5) Http*Base wrapper classes (HttpContextBase, HttpRequestBase, HttpResponseBase, etc) instead of the original Http* classes. This allows you to create mock implementations that inherit the Http*Base classes without an actual HTTP request. This is useful for unit testing, and for overriding standard behaviors (such as route checking).

In ordinary MVC code, the HttpContext, Request, and Response properties will return Http*Wrapper instances that directly wrap the original Http* classes (eg, HttpContextWrapper, which wraps HttpContext). Most MVC developers use the HttpContext and related properties without being aware of any of this redirection.

Until you call Response.RedirectToRoute. This method, which is new to .Net 4.0, redirects the browser to a URL for a route in the new ASP.Net routing engine. Like other HttpResponse methods, HttpResponseBase has its own version of this method for derived classes to override.

However, in .Net 4.0, Microsoft forgot to override this method in the standard HttpResponseWrapper. Therefore, if you call Response.RedirectToRoute in an MVC application (where Response is actually an HttpResponseWrapper), you’ll get a NotImplementedException.

You can see this oversight in the methods list for HttpResponseWrapper. Every method except for RedirectToRoute and RedirectToRoutePermanent are list as (Overrides HttpResponseBase.MethodName().); these methods are listed as (Inherited from HttpResponseBase.MethodName().)

To work around this issue, you can either use the original HttpResponse by writing HttpContext.Current.Response.RedirectToRoute(…) or by calling Response.Redirect instead.

Note that most MVC applications should not call Response.Redirect or Response.RedirectToRoute at all; instead, they should return ActionResults by calling helper methods like return Redirect(…); or return RedirectToAction(…);

In the upcoming ASP.Net 4.5 release, these methods have been properly overridden.

Caller Info Attributes vs. Stack Walking

2011-10-23T23:55:00+00:00

People sometimes wonder why C# 5 needs to add caller info attributes, when this information is already available by using the StackTrace class. In reality, caller info attributes behave rather differently from the StackTrace class, for a number of reasons.

Advantages to Caller Info Attributes

The primary reason to use caller info attributes is that they’re much faster. Stack walking is one of the slowest internal (as opposed to network IO) things you can do in .Net (disclaimer: I haven’t measured). By contrast, caller info attributes have exactly 0 performance penalty. Caller info is resolved at compile-time; the callsite is compiled to pass a string or int literal that was determined by the compiler. Incidentally, this is why C# 5 doesn’t have [CallerType] or [CallerMemberInfo] attributes; the compiler team wasn’t happy with the performance of the result IL and didn’t have time to implement proper caching to make it faster.

Caller info attributes are also more reliable than stack walking. Stack walking will give incorrect results if the JITter decides to inline either your method or its caller. Since caller info attributes are resolved at compile-time, they are immune to inlining. Also, stack walking can only give line number and filename info if the calling assembly’s PDB file is present at runtime, whereas [CallerFileName] and [CallerLineNumber] will always work.

Caller info attributes are also unaffected by compiler transformations. If you do a stack walk from inside a lambda expression, iterator, or async method, you’ll see the compiler-generated method; there is no good way to retrieve the name of the original method. Since caller info attributes are processed by the compiler, [CallerMemberName] should correctly pass the name of the containing method (although I cannot verify this). Note that stack walking will retrieve the correct line number even inside lambda expressions (assuming there is a PDB file)

Advantages To Stack Walking

There are still a couple of reasons to use the StackTrace class. Obviously, if you want to see more than one frame (if you want your caller’s caller), you need to use StackTrace. Also, if you want to support clients in languages that don’t feature caller info attributes, such as C# 4 or F#, you should walk the stack instead.

Stack walking is the only way to find the caller’s type or assembly (or a MemberInfo object), since caller info attributes do not provide this information.

Stack walking is also the only choice for security-related scenarios. It should be obvious that caller info attributes can trivially be spoofed; nothing prevents the caller from passing arbitrary values as the optional parameter. Stack walking, by contrast, happens within the CLR and cannot be spoofed.

Note that there are some other concerns with stack walking for security purposes.

Filename and line number information can be spoofed by providing a modified PDB file
Class and function names are meaningless; your enemy can name his function whatever he wants to. All you should rely on is the assembly name.
There are various ways for an attacker to cause your code to be called indirectly (eg, DynamicInvoking a delegate) so that his assembly isn’t the direct caller. In fact, if the attacker invokes your delegate on a UI thread (by calling BeginInvoke), his assembly won’t show up on the callstack at all. The attacker can also compile a dynamic assembly that calls your function and call into that assembly.
As always, beware of inlining

Subtleties of C# 5’s new [CallerMemberName]

2011-10-18T00:31:00+00:00

UPDATE: Now that the Visual Studio 11 beta has shipped with this feature implemented, I wrote a separate blog post exploring how it actually behaves in these corner cases.

Last time, I explored various pathological code samples in which the [CallerLineNumber] attribute does not have obvious behavior. This time, I’ll cover the last of these new caller info attributes: [CallerMemberName].

The [CallerMemberName] attribute tells the compiler to insert the name of the containing member instead of a parameter’s default value. Unlike [CallerLineNumber] and [CallerFileName], this has no equivalent in C++; since the C / C++ versions of these features are in the preprocessor, they cannot be aware of member names.

Most calls to methods with optional parameters take place within a named method, so the behavior of this attribute is usually obvious. However, there are a couple of places where the exact method name is not so obvious.

If you call a [CallerMemberName] method inside a property or event accessor, what name should the compiler pass? Common sense indicates that it should pass the name of the property or event, since that’s the name you actually see in source code. That would also allow this attribute to be used for raising PropertyChanged events. However, this option doesn’t pass enough information, since it would not be possible to determine whether it was called from the getter or the setter. To expose the maximal amount of information, the compiler should pass the name of the actual method for the accessor – get_Name or set_Name.

I would assume that the compiler only passes the property name, since that is what most people would probably expect.

A less-trivial question arises when such a method is called from a constructor or static constructor. Should the compiler just pass the name of the class, since that’s what the member is named in source code? If so, there would be no way to distinguish between an instance constructor and a static constructor. Should the compiler pass the actual names of the CLR methods (.ctor and .cctor)? If so, there would be no way to tell the class name, which is worse. Should it pass both (ClassName.ctor)? That would expose the maximal amount of information, but wouldn’t match the behavior in other members, which does not include the class name.

On a related note, what about calls to base or this constructors that take [CallerMemberName] arguments? Is that considered part of the class’ constructor, even though the call is lexically scoped outside the constructor? If not, what should it pass?

A further related concern is field initializers. Since field initializers aren’t explicitly in any member, what should the compiler pass if you call a [CallerMemberName] method in a field initializer? I would assume that they’re treated like contructors (or static constructors for static field initializers)

I would assume that a call to a [CallerMemberName] method from within an anonymous method or LINQ query would use the name of the parent method.

The most interesting question concerns attributes. What should happen if you declare your own custom attribute that takes a [CallerMemberName] parameter, then apply the attribute somewhere without specifying the parameter?

If you place the attribute on a parameter, return value, or method, it would make sense for the compiler to pass the name of the method that the attribute was applied to. If you apply the attribute to a type, it might make sense to pass the name of that type. However, there is no obvious choice for attributes applied to a module or assembly.

I suspect that they instead chose to not pass these caller info in default parameters for attribute declarations, and to instead pass the parameters’ declared default values. If so, it would make sense to disallow caller info attributes in attribute constructor parameters. However, this would also prevent them from being used for attributes that are explicitly instantiated in normal code (eg, for global filters in MVC).

Next Time: Caller Info Attributes vs. Stack Walking

Subtleties of C# 5’s new [CallerLineNumber]

2011-10-07T15:20:00+00:00

UPDATE: Now that the Visual Studio 11 beta has shipped with this feature implemented, I wrote a separate blog post exploring how it actually behaves in these corner cases.

This is part 2 in a series about C# 5’s new caller info attributes; see the introduction.

The [CallerLineNumber] attribute tells the compiler to use the line number of the call site instead of the parameter’s default value. This attribute has more corner cases than [CallerFileName]. In particular, unlike the C preprocessor’s __LINE__ macro, the C# compiler inserts the line number of a parsed method call. Therefore, it is not always clear which line a method call expression maps to.

What should this call print:

static class Utils {
    int GetLine([CallerLineNumber] int line = 0) {
        return line;
    }
}

Console.WriteLine(
    Utils
        .
        GetLine
        (
        )
    );

Should it print the line number that the statement started? The line in which the call to GetLine started? The line containing the parentheses for GetLine? What if it’s in a multi-line lambda expression?

There are also a few cases in which methods are called implicitly by the compiler without appearing in source code. What should this code print?

class Funny {
    public Funny Select(Func<object, object> d,
                        [CallerLineNumber]int line = 0) {
        Console.WriteLine(line + ": " + d(d));
        return this;
    }
}

var r = (from x in new Funny() 
         let y = 1 
         select 2);

This code contains two implicit calls to the Select method that don’t have a clear source line (it gets worse for more complicated LINQ queries)

In fact, it is possible to have an implicit method call with no corresponding source code at all.

Consider this code:

class Loggable {
    public Loggable([CallerLineNumber] int line = 0) { }
}
class SomeClass : Loggable { }

The compiler will implicitly generate a constructor for SomeClass that calls the base Loggable constructor with its default parameter value. What line should it pass? In fact, if SomeClass is a partial class that is defined in multiple files, it isn’t even clear what [CallerFileName] should pass.

Also, what should happen in the unlikely case that a [CallerLineNumber] method is called on line 3 billion (which would overflow an int)? (This would be easier to test on Roslyn with a fake stream source) Should it give an integer overflow compile-time error? If [CallerLineNumber] also supports byte and short parameters, this scenario will be more likely to happen in practice.

Next time: [CallerMemberName]

Subtleties of the new Caller Info Attributes in C# 5

2011-10-06T13:02:00+00:00

UPDATE: Now that the Visual Studio 11 beta has shipped with this feature implemented, I wrote a separate blog post exploring how it actually behaves in these corner cases.

C# 5 is all about asynchronous programming. However, in additional to the new async features, the C# team managed to slip in a much simpler feature: Caller Info Attributes.

Since C#’s inception, developers have asked for __LINE__ and __FILE__ macros like those in C and C++. Since C# intentionally does not support macros, these requests have not been answered. Until now.

C# 5 adds these features using attributes and optional parameters. As Anders Hejlsberg presented at //Build/, you can write

public static void Log(string message,
    [CallerFilePath] string file = "",
    [CallerLineNumber] int line = 0,
    [CallerMemberName] string member = "")
{
    Console.WriteLine("{0}:{1} – {2}: {3}", 
                      file, line, member, message);

}

If this method is called in C# 5 without specifying the optional parameters, the compiler will insert the file name, line number, and containing member name instead of the default values. If it’s called in older languages that don’t support optional parameters, those languages will pass the default values, like any other optional method.

These features look trivial at first glance. However, like most features, they are actually more complicated to design in a solid and robust fashion. Here are some of the less obvious issues that the C# team needed to deal with when creating this feature:

Disclaimer: I am basing these posts entirely on logical deduction. I do not have access to a specification or implementation of this feature; all I know is what Anders announced in his //Build/ presentation. However, the C# team would have needed to somehow deal with each of thee issues. Since these attributes are not yet supported by any public CTP, I can’t test my assumptions

To start with, they needed to create new compiler errors if the attribute is applied to an incorrectly-typed parameter, or a non-optional parameter. Creating compiler errors is expensive; they need to be documented, tested, and localized into every language supported by C#.

The attributes should perhaps support nullable types, or parameter types with custom implicit conversions to int or string. (especially long or short)

If a method with these attributes is called in an expression tree literal (a lambda expression converted to an Expression<TDelegate>), the compiler would need to insert ConstantExpressions with the actual line number or other info to pass as the parameter.

In addition to being supported in methods, the attributes should also be supported on parameters for delegate types, allowing you to write

delegate void LinePrinter([CallerLineNumber] int line = 0);

LinePrinter printer = Console.WriteLine;
printer();

Each of the individual attributes also has subtle issues.

[CallerFilePath] seems fairly simple. Any function call must happen in source code, in a source file that has a path. However, it needs to take into account #line directives, so that, for example, it will work as expected in Razor views. I don’t know what it does inside #line hidden, in which there isn’t a source file.

Next time: [CallerLineNumber]

Using a default controller in ASP.Net MVC

2011-09-19T02:01:00+00:00

One common question about ASP.Net MVC is how to make “default” controller.

Most websites will have a Home controller with actions like About, FAQ, Privacy, or similar pages. Ordinarily, these actions can only be accessed through URLs like ~/Home/About. Most people would prefer to put these URLs directly off the root: ~/About, etc.

Unfortunately, there is no obvious way to do that in ASP.Net MVC without making a separate route or controller for each action.

You cannot simply create a route matching "/{action}" and map it to the Home controller, since such a route would match any URL with exactly one term, including URLs meant for other controllers. Since the routing engine is not aware of MVC actions, it doesn’t know that this route should only match actions that actually exist on the controller.

To make it work, we can add a custom route constraint that forces this route to only match URLs that correspond to actual methods on the controller.

To this end, I wrote an extension method that scans a controller for all action methods and adds a route that matches actions in that controller. The code is available at gist.github.com/1225676. It can be used like this:

routes.MapDefaultController<Controllers.HomeController>();

This maps the route "/{action}/{id}" (with id optional) to all actions defined in HomeController. Note that this code ignores custom ActionNameSelectorAttributes. (The built-in [ActionName(…)] is supported)

For additional flexibility, you can also create custom routes that will only match actions in a specific controller. This is useful if you have a single controller with a number of actions that has special route requirements that differ from the rest of your site.

For example:

routes.MapControllerActions<UsersController>(
    name: "User routes",
    url:  "{userName}/{action}"
    defaults: new { action = "Index" }
);

(Note that this example will also match URLs intended for other controllers with the same actions; plan your routes carefully)

XRegExp breaks jQuery Animations

2011-09-15T01:12:00+00:00

Update: This bug was fixed in XRegExp 1.5.1.
However, as far as I know, there are no released versions of SyntaxHighlighter that contain the fix.

XRegExp is an open source JavaScript library that provides an augmented, extensible, cross-browser implementation of regular expressions, including support for additional syntax, flags, and methods.

It’s used by the popular SyntaxHighlighter script, which is in turn used by many websites (including this blog) to display syntax-highlighted source code on the client. Thus, XRegExp has a rather wide usage base.

However, XRegExp conflicts with jQuery. In IE, any page that includes XRegExp and runs a numeric jQuery animation will result in “TypeError: Object doesn't support this property or method”.

Demo (only fails in IE)

This bug is caused by an XRegExp fix for an IE bug in which the exec method doesn't consistently return undefined for nonparticipating capturing groups. The fix, on line 271, assumes that the parameter passed to exec is a string. This behavior violates the ECMAScript standard (section 15.10.6.2), which states that the parameter to exec should be converted to a string before proceeding.

jQuery relies on this behavior in its animate method, which parses a number using a regex to get the decimal portion. (source)

Thus, calling animate with a number after loading XRegExp will fail in IE when XRegExp tries to call slice on a number.

Fortunately, it is very simple to fix XRegExp to convert its argument to a string first:

if (XRegExp) {
    var xExec = RegExp.prototype.exec;
    RegExp.prototype.exec = function(str) {
        if (!str.slice)
            str = String(str);
        return xExec.call(this, str);
    };
}

Here is an updated demo that uses this fix and works even in IE.

Clarifying Boolean Parameters, part 2

2011-09-14T15:30:00+00:00

Part 1 is here

Some languages have better ways to pass boolean parameters. C# 4.0, and all versions of VB, allow parameters to be passed by name. This allows us to write much clearer code:

//C# 4.0:
UpdateLayout(doFullLayout: false)

'VB.Net:
UpdateLayout(doFullLayout:=False)

Without requiring any changes to the function definition, this makes the meaning of the true / false abundantly clear at the call-site.

Javascript offers another interesting alternative. In Javascript, booleans conditions actually check for “truthyness”. The statement if(x) will trigger not just if x is true, but also if x is any “truthy” value, including any object, non-empty string, or non-zero number. Similarly, the expression !x will return false if x is “truthy” and true if x “falsy”.

This means that we can actually use any non-empty string instead of true in Javascript. Note that this will only work if the function checks the value for “truthyness”; it won’t work for code like if (x === true).

Thus, instead of passing true as a boolean, you can pass a string that describes what you’re actually indicating.

For example:

function updatePosition(animate) {
    //Calculate position
    if (animate)
        //...
    else
        //...
}

$(window).resize(function() {
    updatePosition();
});

updatePosition("With animation");

Although this results in much more readable code, it can be difficult to understand for people who aren’t familiar with this trick. If the meaning of the parameter changes, you’ll need to hunt down every place that the function is called and change the string to reflect the new meaning.

Finally, unlike an enum, this does not scale to multiple options. If you need to have more than two options, you should use global variables or objects to simulate an enum, not strings.

Clarifying Boolean Parameters, part 1

2011-09-12T21:31:00+00:00

Have you ever written code like this:

public void UpdateLayout(bool doFullLayout) {
    //Code
    if (doFullLayout) {
        //Expensive code
    }
    //More code
}

This pattern is commonly used when some operation has a “cheap” mode and an “expensive” mode. Other code will have calls like UpdateLayout(false) and UpdateLayout(true) scattered throughout.

The problem is that this isn’t very obvious for people who aren’t familiar with the codebase. If you take a look at a file you’ve never seen before and see calls like UpdateLayout(false) and UpdateLayout(true) scattered, you’ll have no idea what the true / false means.

The simplest solution is to break it out into two methods: UpdateComplexLayout() and UpdateBasicLayout(). However, if the two different layout modes have intertwined code paths (eg, the code before and after the if above), this either won’t be possible or will lead to ugly duplication of code.

One alternative is to use enums:

public enum LayoutUpdateType {
    Basic,
    Full
}

public void UpdateLayout(LayoutUpdateType type) {
    //Code
    if (type == LayoutUpdateType.Full) {
        //Expensive code
    }
    //More code
}

This way, the callsites are much more descriptive: UpdateLayout(LayoutUpdateType.Full). This also makes it easy to add more update modes in the future should the need arise. However, it makes the callsites much more verbose. When used frequently, this pattern can lead a vast proliferation of enum types that are each only used by one method, polluting the namespace and making more important enums harder to notice.

Next time: Cleverer alternatives

C# is not type-safe

2011-09-08T02:12:00+00:00

C# is usually touted as a type-safe language. However, it is not actually fully type-safe!

To examine this claim, we must first provide a strict definition of type-safety.
Wikipedia says:

In computer science, type safety is the extent to which a programming language discourages or prevents type errors. A type error is erroneous or undesirable program behavior caused by a discrepancy between differing data types.

To translate this to C#, full type-safety means that any expression that compiles is guaranteed to work at runtime, without causing any invalid cast errors.

Obviously, the cast (and as) operator is an escape hatch from type safety. It tells the compiler that “I expect this value to actually be of this type, even though you can’t prove it. If I’m wrong, I’ll live with that”. Therefore, to be fully type-safe, it must be impossible to get an InvalidCastException at runtime in C# code that does not contain an explicit cast.

Note that parsing or conversion errors (such as any exception from the Convert class) don’t count. Parsing errors aren’t actually invalid cast errors (instead, they come from unexpected strings), and conversion errors from from cast operations inside the Convert class. Also, null reference exceptions aren’t cast errors.

So, why isn’t C# type-safe?

MSDN says that InvalidCastException is thrown in two conditions:

For a conversion from a Single or a Double to a Decimal, the source value is infinity, Not-a-Number (NaN), or too large to be represented as the destination type.

A failure occurs during an explicit reference conversion.

Both of these conditions can only occur from a cast operation, so it looks like C# is in fact type safe.

Or is it?

IEnumerable numbers = new int[] { 1, 2, 3 };

foreach(string x in numbers) 
    ;

This code compiles (!). Running it results in

InvalidCastException: Unable to cast object of type 'System.Int32' to type 'System.String'.

On the foreach line.

Since we don’t have any explicit cast operations (The implicit conversion from int[] to IEnumerable is an implicit conversion, which is guaranteed to succeed) , this proves that C# is not type-safe.

What happened?

The foreach construct comes from C# 1.0, before generics existed. It worked with untyped collections such as ArrayList or IEnumerable. Therefore, the IEnumerator.Current property that gets assigned to the loop variable would usually be of type object. (In fact, the foreach statement is duck-typed to allow the enumerator to provide a typed Current property, particularly to avoid boxing).

Therefore, you would expect that almost all (non-generic) foreach loops would need to have the loop variable declared as object, since that’s the compile-time type of the items in the collection. Since that would be extremely annoying, the compiler allows you to use any type you want, and will implicitly cast the Current values to the type you declared. Thus, mis-declaring the type results in an InvalidCastException.

Note that if the foreach type isn’t compatible at all with the type of the Current property, you will get a compile-time error (just like (string)42 doesn’t compile). Therefore, if you stick with generic collections, you’re won’t get these runtime errors (unless you declare the foreach as a subtype of the item type).

C# also isn’t type-safe because of array covariance.

string[] strings = new string[1];
object[] arr = strings;
arr[0] = 7;

This code compiles, but throws “ArrayTypeMismatchException: Attempted to access an element as a type incompatible with the array.” at run-time.

As Eric Lippert explains, this feature was added in order to be more compatible with Java.

Delegates vs. Function Pointers, Addendum: Multicast Delegates

2011-08-16T01:02:00+00:00

Until now, I've been focusing on only one of the differences between delegates and function pointers; namely, associated state.
Delegates have one other capability that function pointers do not. A single function pointer can only point to one function. .Net, on the other hand, supports multicast delegates – delegates that point to multiple functions. You can combine two existing delegates using the + operator (or by calling Delegate.Combine) to create a single new delegate instance that points two all of the methods in the original two delegates. This new delegate stores all of the methods from the original two delegates in a private array of delegates called InvocationList (the delegates in this array are ordinary non-multicast delegates that each only point to a single method).

Note that delegates, like strings, are immutable. Adding two delegates together creates a third delegate containing the methods from the first two; the original delegate instances are not affected. For example, writing delegateField += SomeMethod creates a new delegate instance containing the methods originally in delegateField as well as SomeMethod, then stores this new instance in delegateField.

Similarly, the - operator (or Delegate.Remove) will remove the second operand from the first one (again, returning a new delegate instance). If the second operand has multiple methods, all of them will be removed from the final delegate. If some of the methods in the second operand appear multiple times in the original delegate, only the last occurrence of each one will be removed (the one most recently added). The RemoveAll method will remove all occurrences. If all of the methods were removed, it will return null; there is no such thing as an empty delegate instance.

Multicast delegates are not intended to be used with delegates that return values. If you call a non-void delegate that contains multiple methods, it will return the return value of the last method in the delegate. If you want to see the return values of all of the methods, you’ll need to loop over GetInvocationList() and call each delegate individually.

Multicast delegates also don’t play well with the new covariant and contravariant generic delegates in .Net 4.0. You cannot combine two delegates unless their types match exactly, including variant generic parameters.

Function pointers cannot easily be combined the way multicast delegates can. The only way to combine function pointers without cooperation from the code that calls the pointer is to make a function that uses a closure to call all of the function pointers you want to call.

In Javascript, that would look like this:

function combine() {
    var methods = arguments;

    return function() { 
        var retVal;
        for(var i = 0; i < methods.length; i++) 
            retVal = methods[i].apply(this, arguments);
        return retVal;
    };
}

Tracking Event Handler Registrations

2011-07-29T19:58:00+00:00

When working with large .Net applications, it can be useful to find out where event handlers are being registered, especially in an unfamiliar codebase.

In simple cases, you can do this by right-clicking the event definition and clicking Find All References (Shift+F12). This will show you every line of code that adds or removes a handler from the event by name. For field-like (ordinary) events, this will also show you every line of code that raises the event.

However, this isn’t always good enough. Sometimes, event handlers are not added by name. The .Net data-binding infrastructure, as well as the CompositeUI Event Broker service, will add and remove event handlers using reflection, so they won’t be found by Find All References. Similarly, if an event handler is added by an external DLL, Find All References won’t find it.

For these scenarios, you can use a less-obvious trick. As I described last time, adding or removing an event handler actually executes code inside of an accessor method. Like any other code, we can set a breakpoint to see where the code is executed.

For custom events, this is easy. Just add a breakpoint in the add and/or remove accessors and run your program. Whenever a handler is added or removed, the debugger will break into the accessor, and you can look at the callstack to determine where it’s coming from.

However, most events are field-like, and don’t have actual source code in their accessor methods. To set a breakpoint in a field-like event, you need to use a lesser-known feature: function breakpoints (Unfortunately, this feature is not available in Visual Studio Express). You can click Debug, New Breakpoint, Break at Function (Ctrl+D, N) to tell the debugger to pause whenever a specific managed function is executed.

To add a breakpoint at an event accessor, type Namespace.ClassName.add_EventName. To ensure that you entered it correctly, open the Debug, Breakpoints window (Ctrl+D, B) and check that the new breakpoint says break always (currently 0) in the Hit Count column. If it doesn’t say (currently 0), then either the assembly has not been loaded yet or you made a typo in the location (right-click the breakpoint and click Location).

About .Net Events

2011-07-29T02:14:00+00:00

A .Net event actually consists of a pair of accessor methods named add_EventName and remove_EventName. These functions each take a handler delegate, and are expected to add or remove that delegate from the list of event handlers.

In C#, writing public event EventHandler EventName; creates a field-like event. The compiler will automatically generate a private backing field (also a delegate), along with thread-safe accessor methods that add and remove handlers from the backing field (like an auto-implemented property). Within the class that declared the event, EventName refers to this private backing field. Thus, writing EventName(...) in the class calls this field and raises the event (if no handlers have been added, the field will be null).

You can also write custom event accessors to gain full control over how handlers are added to your events. For example, this event will store and trigger handlers in reverse order:

void Main()
{
    ReversedEvent += delegate { Console.WriteLine(1); };
    ReversedEvent += delegate { Console.WriteLine(2); };
    ReversedEvent += delegate { Console.WriteLine(3); };

    OnReversedEvent();
}

protected void OnReversedEvent() {
    if (reversedEvent != null)
        reversedEvent(this, EventArgs.Empty);
}

private EventHandler reversedEvent;
public event EventHandler ReversedEvent {
    add {
        reversedEvent = value + reversedEvent;
    }
    remove {
        reversedEvent -= value;
    }
}

This add accessor uses the non-commutative delegate addition operator to prepend each new handler to the delegate field containing the existing handlers. The raiser method simply calls the combined delegate in the private field. (which is null if there aren’t any handlers)

Note that this code is not thread-safe. If two threads add a handler at the same time, both of them will read the original storage field, add their respective handlers to create a new delegate instance, then write this new delegate back to the field. The thread that writes back to the field last will overwrite the changes made by the other thread, since it never saw the other thread’s handler (this is the same reason that x += y is not thread-safe). The accessors generated by the compiler are threadsafe, either by using lock(this) (C# 3 or earlier) or a lock-free threadsafe implementation (C# 4). For more details, see this series of blog posts.

This example is rather useless. However, there are better reasons to create custom event accessors. WinForms controls store their events in a special EventHandlerList class to save memory. WPF controls create events using the Routed Event system, and store handlers in special storage in DependencyObject. Custom event accessors can also be used to perform validation or logging.

Creating Local Extension Methods

2011-07-27T02:08:00+00:00

Sometimes, it can be useful to make an extension method specifically for a single block of code. Unfortunately, since extension methods cannot appear in nested classes, there is no obvious way to do that.

Instead, you can create a child namespace containing the extension method. In order to limit the extension method’s visibility to a single method, you can put that method in a separate namespace block. This way, you can add a using statement to that namespace alone.

For example:

namespace Company.Project {
    partial class MyClass {
        ...
    }
}
namespace Company.Project {
    using MyClassExtensions;
    namespace MyClassExtensions {
        static class Extensions {
            public static string Name<T>(this T obj) {
                if (default(T) == null && Equals(obj, default(T)))
                    return "(null " + typeof(T) + ")";
                return obj.GetType() + ": " + obj.ToString() 
                     + "{declared as " + typeof(T) + "}";
            }
        }
    }
    partial class MyClass {
        void DoSomething() {
            object x = new DateTime();
            string name = x.Name();
        }
    }
}

Since the using MyClassExtensions statement appears inside the second namespace block, the extension methods are only visible within that block. Code that uses these extension method can appear in this second block, while the rest of the class can go in the original namespace block without the extension methods.

This technique should be avoided where possible, since it leads to confusing and non-obvious code. However, there are situations in which this can make some code much more readable.

Don’t modify other controls during a WPF layout pass

2011-07-26T02:18:00+00:00

Unlike WinForms or native Win32 development, WPF provides a rich layout model which allows developers to easily create complicated UIs that resize to fit their contents or the parent window.

However, when developing custom controls, it can be necessary to layout child controls manually by overriding the MeasureOverride and ArrangeOverride methods. To quote MSDN,

Measure allows a component to determine how much size it would like to take. This is a separate phase from Arrange because there are many situations where a parent element will ask a child to measure several times to determine its optimal position and size. The fact that parent elements ask child elements to measure demonstrates another key philosophy of WPF – size to content. All controls in WPF support the ability to size to the natural size of their content. This makes localization much easier, and allows for dynamic layout of elements as things resize. The Arrange phase allows a parent to position and determine the final size of each child.

Overriding these methods gives your custom control full power over the layout of its child element(s).

Be careful what you do when overriding these methods. Any code in MeasureOverride or ArrangeOverride runs during the WPF layout passes. in these methods, you should not modify any part of the visual tree outside of the control you’re overriding in. If you do, you’ll be changing the visuals between Measure() and Arrange(), which will have unexpected results.

It is safe to modify your own child controls during the layout pass. Before you call Measure() on a child control, its layout pass has not started. Therefore, any changes will be seen by the child’s layout code. Similarly, after you Arrange() a child control, its layout pass is finished, so it is safe to modify again (although you may end up triggering another layout pass to see the changes).

If you do need to modify an outside control during the layout pass, you should call Dispatcher.BeginInvoke() to run code asynchronously during the next message loop. This way, your code will run after the layout pass finishes, and it will be able to safely modify whatever it wants.

Note that Measure() can be called multiple times during a single layout pass (if a parent needs to iteratively determine the best fit for a child).

Delegates vs. Function Pointers, part 5: Javascript

2011-07-25T00:33:00+00:00

This is part 5 in a series about state and function pointers; part 1 is here.

Last time, we saw how C# 2 supports closures by compiling anonymous functions into member functions of a special class that holds local state from the outer function.

Unlike the languages we’ve looked at before, Javascript has had closures baked in to the languages since its inception. My standard example can be achieved very simply in Javascript:

var x = 2;
var numbers = [ 1, 2, 3, 4 ];
var hugeNumbers = numbers.filter(function(n) { return n > x; });

This code uses the Array.filter method, new to Javascript 1.6, to create a new array with those elements from the first array that pass a callback. The function expression passed to filter captures the x variable for use inside the callback.

This looks extremely similar to the C# 2.0 version from last time. However. under the covers, it’s rather different.

Like .Net managed instance methods, all Javascript functions take a hidden this parameter. However, unlike .Net, Javascript does not have delegates. There is no (intrinsic) way to bind an object to the this parameter the way a .Net closed delegate does. Instead, the this parameter comes from the callsite, depending on how the function was called. Therefore, we cannot pass state in the this parameter the way we did in C#.

Instead, all Javascript function expressions capture the variable environment of the scope that they are declared in as a hidden property of the function. Therefore, a function can reference local variables from its declaring scope. Unlike C#, which binds functions to their parent scopes using a field in a separate delegate object that points to the function, Javascript functions have their parent scopes baked in to the functions themselves.

Javascript doesn’t have separate delegate objects that can hold a function and a this parameter. Instead, the value of the this parameter is determined at the call-site, depending on how the function was called. This is a common source of confusion to inexperienced Javascript developers.

To simulate closed delegates, we can make a method that takes a function as well as a target object to call it on, and returns a new function which calls the original function with this equal to the target parameter. That sounds overwhelmingly complicated, but it’s actually not that hard:

function createDelegate(func, target) {
    return function() { 
        return func.apply(target, arguments);
    };
}

var myObject = { name: "Target!"};
function myMethod() {
    return this.name;
}

var delegate = createDelegate(myMethod, myObject);
alert(delegate());

This createDelegate method returns a function expression that captures the func and target parameters, and calls func in the context of target. Instead of storing the target in a property of a Delegate object (like .Net does), this code stores it in the inner function expression’s closure.

Javascript 1.8.5 provides the Function.bind method, which is equivalent to this createDelegate method, with additional capabilities as well. In Chrome, Firefox 4, and IE9, you can write

var myObject = { name: "Target!"};
function myMethod() {
    return this.name;
}

var delegate = myMethod.bind(myObject);
alert(delegate());

For more information, see the MDN documentation.

Delegates vs. Function Pointers, part 4: C# 2.0+

2011-06-19T19:49:00+00:00

This is part 4 in a series about state and function pointers; part 1 is here.

Last time, we saw that it is possible to pass local state with a delegate in C#. However, it involves lots of repetitive single-use classes, leading to ugly code.

To alleviate this tedious task, C# 2 supports anonymous methods, which allow you to embed a function inside another function. This makes my standard example much simpler:

//C# 2.0
int x = 2;
int[] numbers = { 1, 2, 3, 4 };

int[] hugeNumbers = Array.FindAll(
    numbers, 
    delegate(int n) { return n > x; }
);



//C# 3.0
int x = 2;
int[] numbers = { 1, 2, 3, 4 };

IEnumerable<int> hugeNumbers = numbers.Where(n => n > x);

Clearly, this is much simpler than the C# 1.0 version from last time. However, anonymous methods and lambda expressions are compile-time features; the CLR itself is not aware of them. How does this code work? How can an anonymous method use a local variable from its parent scope?

This is an example of a closure – a function bundled together with external variables that the function uses. The C# compiler handles this the same way that I did manually last time in C# 1: it generates a class to hold the function and the variables that it uses, then creates a delegate from the member function in the class. Thus, the local state is passed as the delegate’s this parameter.

To see how the C# compiler implements closures, I’ll use ILSpy to decompile the more-familiar C# 3 version: (I simplified the compiler-generated names for readability)

[CompilerGenerated]
private sealed class ClosureClass {
    public int x;
    public bool Lambda(int n) {
        return n > this.x;
    }
}
private static void Main() {
    ClosureClass closure = new ClosureClass();
    closure.x = 2;
    int[] numbers = { 1, 2, 3, 4 };
    IEnumerable<int> hugeNumbers = numbers.Where(closure.Lambda);
}

The ClosureClass (which was actually named <>c__DisplayClass1) is equivalent to the GreaterThan class from my previous example. It holds the local variables used in the lambda expression. Note that this class replaces the variables – in the original method, instead a local variable named x, the compiler uses the public x field from the ClosureClass. This means that any changes to the variable affect the lambda expression as well.

The lambda expression is compiled into the Lambda method (which was originally named <Main>b__0). It uses the same field to access the local variable, sharing state between the original outer function and its lambda expression.

Next time: Javascript

Open Delegates vs. Closed Delegates

2011-06-14T14:49:00+00:00

.Net supports two kinds of delegates: Open delegates and closed delegates.

When you create a delegate that points to an instance method, the instance that you created it from is stored in the delegate’s Target property. This property is passed as the first parameter to the method that the delegate points to. For instance methods, this is the implicit this parameter; for static methods, it's the method's first parameter. These are called closed delegates, because they close over the first parameter and bring it into the delegate instance.

It is also possible to create open delegates which do not pass a first parameter. Open delegates do not use the Target property; instead, all of the target method’s parameters are passed from the delegate’s formal parameter list, including the first parameter. Therefore, an open delegate pointing to a given method must have one parameter more than a closed delegate pointing to the same method. Open delegates are usually used to point to static methods. When you make a delegate pointing to a static method, you (generally) don't want the delegate to hold a first parameter for the method.

In addition to these two normal cases, it is also possible (in .Net 2.0 and later) to create open delegates for instance methods and to create closed delegates for static methods. With one exception, C# doesn’t have any syntactical support for these unusual delegates, so they can only be created by calling the CreateDelegate method.

Open delegates by calling the CreateDelegate overload that doesn’t take a target parameter. Before .Net 2.0, this function could only be called with a static method. In .Net 2.0, you can call this function with an instance method to create an open delegate. Such a delegate will call use its first parameter as this instead of its Target field.

As a concrete example, consider the String.ToUpperInvariant() method. Ordinarily, this method takes no parameters, and operates on the string it’s called on. An open delegate pointing to this instance method would take a single string parameter, and call the method on that parameter.

For example:

Func<string> closed = new Func<string>("a".ToUpperInvariant);
Func<string, string> open = (Func<string, string>)
    Delegate.CreateDelegate(
        typeof(Func<string, string>),
        typeof(string).GetMethod("ToUpperInvariant")
    );

closed();     //Returns "A"
open("abc");  //Returns "ABC"

Closed delegates are created by calling the CreateDelegate overload that takes a target parameter. In .Net 2.0, this can be called with a static method and an instance of that method’s first argument type to create a closed delegate that calls the method with the given target as its first parameter. Closed delegates curry the first parameter from the target method. For example:

Func<object, bool> deepThought = (Func<object, bool>)
    Delegate.CreateDelegate(
        typeof(Func<object, bool>),
        2,
        typeof(object).GetMethod("Equals", BindingFlags.Static | BindingFlags.Public)
    );

This code curries the static Object.Equals method to create a delegate that calls Equals with 2 and the delegate’s single parameter). It’s equivalent to x => Object.Equals(2, x). Note that since the method is (generally) not a member of the target object’s type, we need to pass an actual MethodInfo instance; a name alone isn’t good enough.

Note that you cannot create a closed delegate from a static method whose first parameter is a value type, because, unlike all instance methods, static delegates that take value types receive their parameters by value, not as a reference. For more details, see here and here.

C# 3 added limited syntactical support for creating closed delegates. You can create a delegate from an extension method as if it were an instance method on the type it extends. For example:

var allNumbers = Enumerable.Range(1, Int32.MaxValue);
Func<int, IEnumerable<int>> countTo = allNumbers.Take;

This code creates an IEnumerable<int> containing all positive integers, then creates a closed delegate that curries this sequence into the static Enumerable.Take<T>(IEnumerable<T>) method.

Except for extension methods, open instance delegates and closed static delegates are rarely used in actual code. However, it is important to understand how ordinary open and closed delegates work, and where the target object ends up for instance delegates.

Delegates vs. Function Pointers, part 3: C# 1.0

2011-06-13T21:29:00+00:00

This is part 3 in a series about state and function pointers; part 1 is here.

Last time, we saw that it is impossible to bundle context along with a function pointer in C.

In C#, it is possible to fully achieve my standard example. In order to explain how this works behind the scenes, I will limit this post to C# 1.0 and not use a lambda expression. This also means no LINQ, generics, or extension methods, so I will, once again, need to write the filter method myself.

delegate bool IntFilter(int num);

static ArrayList Filter(IEnumerable source, IntFilter filter) {
    ArrayList retVal = new ArrayList();

    foreach(int i in source)
        if (filter(i))
            retVal.Add(i);
            
    return retVal;
}

class GreaterThan {
    public GreaterThan(int b) {
        this.bound = b;
    }
    int bound;
    public bool Passes(int num) {
        return num > bound;
    }
}

int x = 2;
int[] numbers = { 1, 2, 3, 4 };
Filter(numbers, new IntFilter(new GreaterThan(x).Passes));

Please excuse the ugly code; in order to be true to C# 1.0, I can’t just write an iterator, and I can’t implicitly create the delegate.

This code creates a class called GreaterThan that holds the Passes method and the state used by the method. I create a delegate to pass to Filter out of the Passes method for an instance of the GreaterThan class from the local variable.

To understand how this works, we need to delve further into delegates. .Net delegates are more than just type-safe function pointers. Unlike the function pointers we've looked at, delegates include state – the Target property. This property stores the object to pass as the hidden this parameter to the method (It's actually somewhat more complicated than that; I will describe open and closed delegates at a later date). My example creates a delegate instance in which the Target property points to the GreaterThan instance. When I call the delegate, the Passes method is called on the correct instance from the Target property, so it can read the bound field.

Next time, we'll see how the C# 2.0+ compilers generate all this boilerplate for you using anonymous methods and lambda expressions.

Delegates vs. Function Pointers, part 2: C

2011-06-01T21:08:00+00:00

This is part 2 in a series about state and function pointers; part 1 is here.

Unlike most other languages, it is not possible to include any form of state in a function pointer in C. Therefore, it is impossible to fully implement closures in C without the cooperation of the call-site and/or the compiler.

To illustrate what this means in practice, I will refer to my standard example, using a filter function to find all elements in an array that are greater than x. Since C doesn’t have any such function, I’ll write one myself, inspired by qsort:

void* filter(void* base, 
             size_t count, 
             size_t elemSize, 
             int (*predicate)(const void *)) {
    //Magic...
}

With this function, one can easily find all elements that are greater than some constant:

int predicate(const void* item) {
        return *(int*)item > 2;
}

int arr[] = { 1, 2, 3, 4 };
filter(arr, 4, sizeof(int), &predicate);

However, if we want to replace the hard-coded 2 with a local variable, it becomes more complicated. Since function pointers, unlike delegates, do not have state, there is no simple way to pass the local variable to the function.

The most obvious answer is to use a global variable:

static int minimum;
int predicate(const void* item) {
        return *(int*)item > minimum;
}

int arr[] = { 1, 2, 3, 4 };
minimum = x;
filter(arr, 4, sizeof(int), &predicate);

However, this code will fail horribly if it’s ever run on multiple threads. In simple cases, you can work around that by storing the value in a global thread-local variable (or in fiber-local storage for code that uses fibers). Because each thread will have its own global, the threads won’t conflict with each-other. Obviously, this won’t work if the callback might be run on a different thread.

However, even if everything is running on one thread, this still isn’t enough if the code can be re-entrant, or if the callback might be called after the original function call finishes. This technique assumes that exactly one callback (from a single call to the function that accepted the callback) will be used at any given time. If the callback might be invoked later (eg, a timer, or a UI handler), this method will break down, since all of the callbacks will share the same global.

In these more complicated cases, one can only pass any state with the cooperation of the original function. In this (overly simple) example, the filter method can be modified to take a context parameter and pass it to the predicate:

void* filter(void* base, 
             size_t count,
             size_t elemSize, 
             int (*predicate)(const void *, const void *), 
             const void* context) {
        //More magic...
}
int predicate(const void* item, const void* context) {
        return *(int*)item > *(int*)context;
}

int arr[] = { 1, 2, 3, 4 };
filter(arr, 4, sizeof(int), &predicate, &x);

To pass around more complicated state, one could pass a pointer to a struct as the context.

Next Time: C# 1.0

Delegates vs. Function Pointers, part 1

2011-06-01T20:21:00+00:00

Most languages – with the unfortunate exception of Java – allow functions to be passed around as variables. C has function pointers, .Net has delegates, and Javascript and most functional programming languages treat functions as first class objects.

There is a fundamental difference between C-style function pointers vs. delegates or function objects. Pure function pointers cannot hold any state other than the function itself. In contrast, delegates and function objects do store additional state that the function can use.

To illustrate this difference, I will use a simple example. Most programming environments have a filter function that takes a collection and a predicate (a function that takes an item from the collection and returns a boolean), and returns a new collection with those items from the original collection that match the predicate. In C#, this is the LINQ Where method; in Javascript, it’s the Array.filter method (introduced by Javascript 1.6).

Given a list of numbers, you can use such a method to find all numbers in the list that are greater than a local variable x. However, in order to do that, the predicate have access to x. In subsequent posts, I’ll explore how this can be done in different languages.

Html.ForEach in Razor

2011-05-11T02:56:00+00:00

Many people write ForEach extension methods for MVC WebForms views, which take a sequence and a delegate to turn each item in the sequence into HTML.

For example:

public static void ForEach<T>(this HtmlHelper html,
                              IEnumerable<T> set, 
                              Action<T> htmlWriter) {
    foreach (var item in set) {
        htmlWriter(item);
    }
}

(The unused html parameter allows it to be called as an extension method like other HTML helpers)

This code can be called like this:

<ul>
    <% Html.ForEach(
            Enumerable.Range(1, 10),
            item => { %> <li><%= item %></li> <% }
    ); %>
</ul>

This code creates a lambda expression that writes markup to the page’s response stream by ending the code block inside the lambda body. Neither the lambda expression nor the ForEach helper itself return anything; they both write directly to the response.

The List<T>.ForEach method can be called exactly the same way.

In Razor views, this method cannot easily be called directly, since Razor pages cannot put markup inside of expressions. You can use a workaround by creating an inline helper and calling it immediately, but it would be better to rewrite the ForEach method to take an inline helper directly.

The naïve way to do that is like this:

public static IHtmlString ForEachSimple<T>(
        this HtmlHelper html,
        IEnumerable<T> set,
        Func<T, HelperResult> htmlCreator
    ) {
    return new HtmlString(String.Concat(set.Select(htmlCreator)));
}

The htmlCreator delegate, which can be passed as an inline helper, returns a HelperResult object containing the markup generated for an item.

This code uses LINQ to call htmlCreator on each item in the set (the Select call), then calls String.Concat to combine them all into one giant string. (String.Concat will call ToString on each HelperResult, which will return the generated markup) We could also call String.Join to put a separator, such as a newline, between every two items.

Finally, it returns an HtmlString to prevent Razor from escaping the returned HTML.

It’s equivalent to the following code using a StringBuilder (this is what String.Concat does internally)

var builder = new StringBuilder();
foreach (var item in set) {
    HelperResult result = htmlCreator(item);
    builder.Append(result.ToString());
}
return new HtmlString(builder.ToString());

This method can be called like this:

<ul>
    @Html.ForEachSimple(
        Enumerable.Range(1, 10),
        @<li>@item</li>
    );
</ul>

The problem with this approach is that it combines all of the content into a giant string. If there are a large number of items, or if each item will have a large amount of markup, this can become (a little bit) slow. It would be better to write each item directly to the caller’s response stream, without assembling any giant strings. This is where HelperResult comes in.

The HelperResult class allows its caller to pass a TextWriter to the WriteTo method, and the helper delegate will write directly to this TextWriter. I can take advantage of this to write a ForEach extension that doesn’t build any strings, by returning a HelperResult instead of a regular IHtmlString.

public static HelperResult ForEachFast<T>(
        this HtmlHelper html,
        IEnumerable<T> set,
        Func<T, HelperResult> htmlCreator
    ) {
    return new HelperResult(tw => {
        foreach (var item in set) {
            htmlCreator(item).WriteTo(tw);
        }
    });
}

This version creates a HelperResult with a delegate that writes each of its items in turn to the TextWriter.

Creating Markup Actions in Razor

2011-05-02T18:06:00+00:00

Razor’s inline helpers allow you to create lambda expression that return markup (as a HelperResult). However, there is no simple way to create a lambda expression that writes HTML directly to the page (instead of returning it).

In ASPX pages, one can simply put the beginning and end of a lambda expression in expression blocks, then put markup in the middle.

For example, this code creates a delegate that writes a <b> tag to the page:

<%
    Action pageWriter = () => {%><b>I'm from a lambda!</b><%};
    pageWriter();
    pageWriter();
    pageWriter();
%>

Calling the pageWriter delegate will write directly to the HTTP response stream.

By contrast, Razor inline expressions return their markup. To do this in a Razor page, one would write

@{
    Func<object, HelperResult> htmlMaker
         = @<b>I'm from a lambda!</b>;
    @htmlMaker(null)    //Note @ sign
    @htmlMaker(null)    //Note @ sign
    @htmlMaker(null)    //Note @ sign
}

Calling htmlMaker without an @ sign will return HTML, but won’t write anything to the page.

When working with libraries designed for ASPX pages, it can be necessary to create ASPX-style inline helpers that write to the page instead of returning a value. You can do that by creating an inline helper lambda and passing it to the Write method:

@{
    Action pageWriter = () => Write(new Func<object, HelperResult>(
         @<b>I'm from a lambda!</b>
    )(null));

    pageWriter();
    pageWriter();
    pageWriter();
}

Like the ASPX version, pageWriter now writes directly to the page and does not return anything.

This code can be made simpler by wrapping it in a separate method:

Action MakeAction(Func<object, HelperResult> inlineHelper) {
    return () => Write(inlineHelper(null));
}

This method takes an inline helper and returns an Action that writes the helper’s output to the page. Since it needs to call the page’s Write method (to use the page’s output stream), this method must be defined in the WebPageBase instance, either in an @functions block or in a common base class.

Any code in the inline helper will execute each time the resulting Action is called.

It can be called like this:

@{
    Action pageWriter2 = MakeAction(@<b>I'm from a lambda!</b>);

    pageWriter();
    pageWriter();
    pageWriter();
}

This code is equivalent to the previous sample, but it’s much simpler.

One can also write write a version that takes a parameter:

Action<T> MakeAction<T>(Func<T, HelperResult> inlineHelper) {
    return param => Write(inlineHelper(param));
}

Note that the type parameter must be specified explicitly; C# does not infer type parameters from the method’s return type.

Dissecting Razor, part 9: Inline Helpers

2011-04-29T14:08:00+00:00

In addition to normal and static helpers, Razor supports inline helpers, (also known as templates), which allow you to create lambda expressions that return markup.

An inline helper is created by writing @<tag>Content</tag> as an expression. This creates a lambda expression that takes a parameter called item and returns a HelperResult object containing the markup.

Inline helpers are used to create functions that take markup as parameters. For example, you might make an IfLoggedOn helper that displays content if there is a logged-in user, but shows a login link to anonymous users. To pass the content to the helper, you can use an inline helper:

@helper IfLoggedOn(Func<MembershipUser, HelperResult> content) {
    var currentUser = Membership.GetUser();
    if (currentUser == null) {
        <i>
            To use this content, you must 
            <a href="@Href("~/Login")">log in</a>.
        </i>
    } else {
        @content(currentUser)
    }
}

You can call this method with an inline helper like this:

@IfLoggedOn(
    @<form action="@Href("Add-Comment")" method="post">
        @Html.TextArea("Comment", "Type a comment")
        <input type="submit" name="Submit" value="Add Comment" />
    </form>
)

The @content(currentUser) call in helper method is translated by the Razor compiler into a call to the overload of the Write method that takes a HelperResult (returned from the delegate). This overload writes the content of the HelperResult to the page without escaping it (Just like a call to a normal helper method). Functions that take inline helpers can also get the text rendered by the helper by calling ToString() on the HelperResult.

The call to the helper method is compiled to the following C# code:

Write(IfLoggedOn(
item => new System.Web.WebPages.HelperResult(__razor_template_writer => {

    WriteLiteralTo(@__razor_template_writer, "    ");
    WriteLiteralTo(@__razor_template_writer, "<form action=\"");

    WriteTo(@__razor_template_writer, Href("Add-Comment"));

    WriteLiteralTo(@__razor_template_writer, "\" method=\"post\">\r\n        ");

    WriteTo(@__razor_template_writer, Html.TextArea("Comment", "Type a comment"));

    WriteLiteralTo(@__razor_template_writer, "\r\n   ...  </form>\r\n");

})));

The IsLoggedOn method is passed a lambda expression that takes an item parameter and returns a new HelperResult. The HelperResult is constructed exactly like a normal helper, except that it’s in a lambda expression instead of a normal method.

Like normal helpers, then markup inside inline helpers can contain arbitrary code blocks and code nuggets. However, inline helpers cannot be nested (since that would create conflicting parameter names)

The item parameter is implicitly typed based on the delegate type that the helper is being passed as (just like normal lambda expressions). This allows inline helpers to accept a parameter:

@IfLoggedOn(@<text>Welcome, @item.UserName!</text>)

(The special <text> tag is used to pass markup without creating an actual HTML tag)

Next Time: Sections

Dissecting Razor, part 8: Static Helpers

2011-03-08T16:41:00+00:00

Razor helpers can be extremely useful, but they are designed to be used only by the page that created them.

To create reusable helpers, you can create CSHTML pages in the App_Code directory. The WebRazorHostFactory will check for pages in App_Code and create a WebCodeRazorHost instead of the normal WebPageRazorHost.

This happens before the virtual CreateHost method; in order to change this behavior, one must create an inherited RazorBuildProvider and override the CreateHost method; for more details, see the source for more details.

The WebCodeRazorHost compiles helper methods as public static methods by setting the RazorEngineHost.StaticHelpers property to true. It also overrides PostProcessGeneratedCode to remove the Execute() method and make the Application property static.

Static helper pages inherit the HelperPage class, a very stripped-down base class which contains the static WriteTo methods used by the helpers and some static members that expose the Model, Html, and other members from the currently executing page (using the WebPageContext.Current.Page property).

Because the generated Execute() method is removed, all content outside helpers and @functions blocks is not seen by the compiler. It is seen by the Razor parser, so it must contain valid Razor syntax (eg, no stray @ characters).

Here is an example:

@helper Menu(params string[][] items) {
    <ul>
        @foreach (var pair in items) {
            <li><a href="@Href(pair[1])">@pair[0]</a></li>
        }
    </ul>
}

This text and @code is not compiled.

This helper can then be called from any other code by writing PageName.Menu(...), where PageName is the filename of the CHSTML page in App_Code. In Razor pages, it can be called like any other helper. In normal C# code, it returns a HelperResult instance; call ToString() or WriteTo(TextWriter) to get the HTML source.

For example: (assuming that the helper is defined in ~/App_Code/Helpers.cshtml)

@Helpers.Menu(
    new[] { "Home",    "~/"        },
    new[] { "About",   "~/About"   },
    new[] { "Contact", "~/Contact" }
)

In order to see the source, I need to insert a #error directive inside the helper block; putting it in the page itself will have no effect, since the page is not compiled. Since the contents of the helper block are code, not markup, I don’t need to wrap it in @{ ... }.

The above helper is transformed into the following C#: (As usual, @line directives have been removed)

public class Helpers : System.Web.WebPages.HelperPage {
    public static System.Web.WebPages.HelperResult Menu(params string[][] items) {
        return new System.Web.WebPages.HelperResult(__razor_helper_writer => {

            WriteLiteralTo(@__razor_helper_writer, "    <ul>\r\n");
            foreach (var pair in items) {
                WriteLiteralTo(@__razor_helper_writer, "            <li><a href=\"");
                WriteTo(@__razor_helper_writer, Href(pair[1]));
                WriteLiteralTo(@__razor_helper_writer, "\">");
                WriteTo(@__razor_helper_writer, pair[0]);
                WriteLiteralTo(@__razor_helper_writer, "</a></li>\r\n");
            }
            WriteLiteralTo(@__razor_helper_writer, "    </ul>\r\n");
#error
        });
    }
    public Helpers() {
    }
    protected static System.Web.HttpApplication ApplicationInstance {
        get {
            return ((System.Web.HttpApplication)(Context.ApplicationInstance));
        }
    }
}

Note that the page-level content does not show up anywhere. The helper itself is compiled exactly like a normal helper, except that it’s static.

Although there aren’t any in this example, @functions blocks will also be emitted normally into the generated source.

Next Time: Inline Helpers

Boot Camp just got better

2011-03-06T16:33:00+00:00

The Boot Camp drivers in the latest generation of MacBook Pros now expose more of the laptop’s hardware to Windows.

This means that Windows can now adjust the screen brightness in the Windows Mobility Center, and when you plug in or unplug the laptop.

The new drivers also expose the laptop’s ambient light sensor via Windows 7’s sensor platform, allowing the screen brightness to be adjusted automatically, and allowing 3rd-party programs to read the brightness.

Dissecting Razor, part 7: Helpers

2011-02-28T16:08:00+00:00

We’ll continue our trek into Razor’s class-level features with helpers.

Helpers are one of Razor’s unique features. They encapsulate blocks of HTML and server-side logic into reusable page-level methods.

You can define a helper by writing @helper MethodName(parameters) { ... }. Inside the code block, you can put any markup or server-side code. The contents of a helper are parsed as a code block (like the contents of a loop or if block), so any non-HTML-like markup must be surrounded by <text> tags or prefixed by the @: escape.

Here is a simple example:

<!DOCTYPE html>
<html>
    <body>
        @helper NumberRow(int num) {
            var square = num * num;
            <tr>
                <td>@num</td>
                <td>@square</td>
                <td>@(num % 2 == 0 ? "Even" : "Odd")</td>
            </tr>
        }
        <table>
            <thead>
                <tr>
                    <th>Number</th>
                    <th>Square</th>
                    <th>Eveness</th>
                </tr>
            </thead>
            <tbody>
                @for (int i = 0; i < 10; i++) {
                    @NumberRow(i)
                }
            </tbody>
        </table>
    </body>
</html>

Note that code statements (such as the square declaration) can go directly inside the helper body without being wrapped in code blocks – the direct contents of the helper is a code block, not markup. Like any other code block, HTML-like markup is automatically treated as markup instead of code.

Like @functions blocks, helper methods can go anywhere in the source file; the physical location of the block is ignored.

Here is the generated source for the above example: (with blank lines and #line directives stripped for clarity)

public class _Page_Razor_Helpers_cshtml : System.Web.WebPages.WebPage {

    public System.Web.WebPages.HelperResult NumberRow(int num) {
        return new System.Web.WebPages.HelperResult(__razor_helper_writer => {

            var square = num * num;

            WriteLiteralTo(@__razor_helper_writer, "            <tr>\r\n                <td>");
            WriteTo(@__razor_helper_writer, num);

            WriteLiteralTo(@__razor_helper_writer, "</td>\r\n                <td>");
            WriteTo(@__razor_helper_writer, square);

            WriteLiteralTo(@__razor_helper_writer, "</td>\r\n                <td>");
            WriteTo(@__razor_helper_writer, num % 2 == 0 ? "Even" : "Odd");

            WriteLiteralTo(@__razor_helper_writer, "</td>\r\n            </tr>\r\n");

        });
    }
    public _Page_Razor_Helpers_cshtml() {
    }
    protected ASP.global_asax ApplicationInstance {
        get {
            return ((ASP.global_asax)(Context.ApplicationInstance));
        }
    }
    public override void Execute() {
        WriteLiteral("<!DOCTYPE html>\r\n<html>\r\n    <body>\r\n        ");
        WriteLiteral("\r\n        <table>\r\n            <thead>\r\n                <tr>\r\n                   " +
        " <th>Number</th>\r\n                    <th>Square</th>\r\n                    <th>E" +
        "veness</th>\r\n                </tr>\r\n            </thead>\r\n            <tbody>\r\n");

        for (int i = 0; i < 10; i++) {
            Write(NumberRow(i));
        }

        WriteLiteral("            </tbody>\r\n        </table>\r\n    </body>\r\n</html>\r\n");
#error

    }
}

Helpers are compiled as class-level methods that take a parameter set and return a System.Web.WebPages.HelperResult. (This class name is configured by the RazorHostFactory)

Notice the the contents of the helper method are inside a lambda expression that takes a parameter named __razor_helper_writer. This construction allows the helper to write directly to the HTTP response stream instead of assembling a giant string and then writing the string all at once.

The HelperResult constructor takes an Action<TextWriter> which contains the contents of the helper block. The class implements IHtmlString and calls the action from the constructor to generate HTML. However, under normal circumstances, this IHtmlString implementation is never called.

Calls to helper methods (@NumberRow(i)) are passed to the Write(HelperResult) overload. This overload calls HelperResult.WriteTo(writer), which passes the page’s TextWriter directly to the helper’s lambda expression. Thus, the lambda expression can write directly to the page’s output stream, without passing the output as a parameter to the helper method.

Looking inside the helper, we see that all content is passed to WriteTo and WriteLiteralTo methods, as opposed to the Write and WriteLiteral methods used by the rest of the page.

Helper methods cannot call the normal Write* methods since they aren’t necessarily writing to the current output (even though they usually do). Therefore, they call these Write*To methods, which accept the TextWriter as a parameter. These static methods are inherited from the WebPageExecutingBase class; their names are also configured by the RazorHostFactory. The @ in the parameter is a little-used C# syntactictal feature that allows keywords to be used as identifiers; it has nothing to do with Razor’s use of the @ character.

Since helpers are compiled as normal methods, they can do almost anything that a normal method can. However, because their contents are compiled inside a lambda expression, they have some limitations. For example, helpers cannot use ref or out parameters, since they cannot be used inside lambda expressions. Helpers can take params arrays or optional parameters.

Also, Razor’s C# code parser doesn’t support generic helper methods, although there is no reason that they couldn’t be supported in a later release.

The VB.Net code parser also doesn’t explicitly support generic helpers. However, because VB.Net generics use parentheses, a VB.Net helper can declare a generic type parameter instead of a normal parameter list.

For example:

<!DOCTYPE html>
<html>

    <body>
        @Helper PrintType(Of T)
            @GetType(T)
        End Helper
        @PrintType(Of Dictionary(Of Integer, List(Of String)))()
    </body>

</html>

This trick is not very useful.

Next Time: Static Helpers

Dissecting Razor, part 6: Function Blocks

2011-02-18T02:46:00+00:00

After looking at how Razor’s Execute() method is generated, we will turn to class-level features.

C# Razor pages can define class members inside of @functions { ... } blocks. These are Razor’s equivalent of <script runat="server"> blocks in ASPX pages. VBHTML pages use @Functions ... End Functions instead.

Functions blocks are emitted directly into top of the generated class, regardless of their location in the original source. Unlike code blocks, function blocks cannot contain markup.

Here is a simple example:

<!DOCTYPE html>
<html>
    <body>
        @functions{
            public int GetPageLength() {
                //Don't try this in production.
                return ((StringWriter)this.Output).ToString().Length;
            }
        }
        @GetPageLength() characters have been written so far.
    </body>
</html>
@{ #error }

Note that functions blocks can be defined anywhere, even in the middle of the markup. The location of the block is totally irrelevant.

Here is the generated C# source, with a comment indicating where the block used to be.

public class _Page_Razor_Functions_cshtml : System.Web.WebPages.WebPage {

#line hidden
#line 4 "...\Functions.cshtml"

    public int GetPageLength() {
        //Don't try this in production.
        return ((StringWriter)this.Output).ToString().Length;
    }

#line default
#line hidden

    public _Page_Razor_Functions_cshtml() {
    }

    protected ASP.global_asax ApplicationInstance {
        get {
            return ((ASP.global_asax)(Context.ApplicationInstance));
        }
    }

    public override void Execute() {
        WriteLiteral("<!DOCTYPE html>\r\n<html>\r\n    <body>\r\n        ");
//Here was the functions block
        WriteLiteral("\r\n        ");

#line 10 "...\Functions.cshtml"
        Write(GetPageLength());
#line default
#line hidden
        WriteLiteral(" characters have been written so far.\r\n    </body>\r\n</html>\r\n");

#line 13 "...\Functions.cshtml"
#error

#line default
#line hidden
        WriteLiteral("     ");
    }
}

The contents of the functions block is inserted at the very top of the class, with the familiar #line directives to pretend that it comes from the CSHTML.

Notice that none of the whitespace around the functions block is stripped; you can see the newline and indentation in the @functions line in the first WriteLiteral string, and the newline and indentation after the closing } in the second WriteLiteral string.

Sure enough, the rendered HTML contains an extra blank like:

<!DOCTYPE html>
<html>
    <body>
        
        55 characters have been written so far.
    </body>
</html>

This means that putting a functions block before the <!DOCTYPE> will cause the HTML to start with an ugly blank line. Therefore, it’s best to put functions blocks at the end of the source, where the blank lines won’t matter.

Next Time: Helpers

Dissecting Razor, part 5: Use the Source, Luke

2011-02-16T16:54:00+00:00

Last time, we saw how basic Razor constructs are translated into C#.

We can see the generated class by adding @{ #error } to the page source. This creates a compiler error in the Execute method, and the resulting Yellow Screen of Death contains a Show Complete Compilation Source: link which will show the generated C# class.

Let’s start with a very simple page:

<!DOCTYPE html>
<html>
    <body>
        1 + 2 = @(1 + 2)<br />
        @{ var source = "<b>bold &amp; fancy</b>"; }
        <code>@source</code> is rendered as
        @(new HtmlString(source))
    </body>
</html>
@{ #error }

This page is rendered like this: (after removing @{ #error })

<!DOCTYPE html>
<html>
    <body>
        1 + 2 = 3<br />
        <code>&lt;b&gt;bold &amp;amp; fancy&lt;/b&gt;</code> is rendered as
        <b>bold &amp; fancy</b>
    </body>
</html>

As expected, the expression @source is automatically escaped. Also notice that the newline and indentation around the code block (@{ var ... }) was not rendered – the Razor parser strips all whitespace surrounding code blocks. This is a welcome improvement over the ASPX view engine.

Now let’s look at how this HTML is generated. This page is transformed into the following C# source:

namespace ASP {
    using System;
    using System.Collections.Generic;
    using System.IO;
    using System.Linq;
    using System.Net;
    using System.Web;
    using System.Web.Helpers;
    using System.Web.Security;
    using System.Web.UI;
    using System.Web.WebPages;
    using System.Web.WebPages.Html;
    using WebMatrix.Data;
    using WebMatrix.WebData;
    public class _Page_Razor_SimplePage_cshtml : System.Web.WebPages.WebPage {

#line hidden
        public _Page_Razor_WriteTo_cshtml() {
        }

        protected ASP.global_asax ApplicationInstance {
            get {
                return ((ASP.global_asax)(Context.ApplicationInstance));
            }
        }

        public override void Execute() {
            WriteLiteral("<!DOCTYPE html>\r\n<html>\r\n    <body>\r\n        1 + 2 = ");

#line 4 "...\SimplePage.cshtml"
            Write(1 + 2);
#line default
#line hidden
            WriteLiteral("<br />\r\n");

#line 5 "...\SimplePage.cshtml"
            var source = "<b>bold &amp; fancy</b>";
#line default
#line hidden
            WriteLiteral("        <code>");

#line 6 "...\SimplePage.cshtml"
            Write(source);
#line default
#line hidden
            WriteLiteral("</code> is rendered as\r\n        ");

#line 7 "...\SimplePage.cshtml"
            Write(new HtmlString(source));
#line default
#line hidden
            WriteLiteral("\r\n    </body>\r\n</html>\r\n");

#line 10 "...\SimplePage.cshtml"
#error
#line default
#line hidden

        }
    }
}

The WebPageRazorEngineHost injects the ApplicationInstance property into the CodeDOM tree; this property allows code in the page to access any custom properties in Global.asax.

As mentioned earlier, the page source is compiled into the Execute() method.

It uses #line directives to pretend that its code is actually in the CSHTML page. This means that code or line numbers appearing in error pages come from the original CSHTML source, making the code easier to find when debugging. The #line hidden directives indicate generated source that did not come from actual code in the CSHTML.

As mentioned last time, literal HTML source is passed to the WriteLiteral method, which is inherited from the WebPageBase class. This method writes its argument to the current output stream (which can vary when making sections). These calls are wrapped in #line hidden because they come from literal text, not code.

The two code blocks (the variable declaration and the #error directive) are copied straight into Execute(), wrapped in #line directives that map them to the actual code lines in the CSHTML.

The code nuggets are passed to the Write method, and are similarly wrapped in #line directives.

Here is a more sophisticated Razor page:

<!DOCTYPE html>
<html>
    <body>
        @{ const int count = 10; }
        <table>
            @for (int i = 0; i < count; i++) {
                <tr>
                    <td>@i</td>
                    <td>@(i * i)</td>
                </tr>
            }
        </table>
    </body>
</html>
@{ #error }

The @for loop is a code block in the form of a control flow statement. Razor’s C# parser is aware of C# control flow structures and parses them as code blocks. (The VB parser does the same thing)

Here is the generated Execute() method, with #line directives removed for clarity:

public override void Execute() {
    WriteLiteral("<!DOCTYPE html>\r\n<html>\r\n    <body>\r\n");

    const int count = 10;
    WriteLiteral("        <table>\r\n");

    for (int i = 0; i < count; i++) {
        WriteLiteral("                <tr>\r\n                    <td>");

        Write(i);
        WriteLiteral("</td>\r\n                    <td>");

        Write(i * i);
        WriteLiteral("</td>\r\n                </tr>\r\n");

    }
    WriteLiteral("        </table>\r\n    </body>\r\n</html>\r\n");
#error
}

Here too, we see that all contiguous chunks of literal HTML are passed to WriteLiteral.

This example has two code blocks – the const declaration and the loop. The for loop code block has HTML inside of it – any HTML markup inside a code block is parsed as normal HTML and passed to WriteLiteral.

Next Time: Function blocks

Dissecting Razor, part 4: Anatomy of a Razor Page

2011-02-09T14:08:00+00:00

After looking at the various assemblies in the WebPages framework, we will drill into the inner workings of Razor pages.

Razor Side

An ordinary CSHTML page is transformed into a class which inherits the WebPage class. The generator overrides the abstract Execute() method from the to render the page to the HTTP response stream.

Except for class-level directives and constructs (which will be discussed later), all ordinary content in a Razor page end up in the Execute method.

There are three types of normal content: Literals, Code Blocks, and Code Nuggets.

Literals include any normal text. Razor compiles literal text into calls to the WriteLiteral method, with the text as a (correctly-escaped) string parameter. Razor expects this method to write its parameter to the page.

Code Blocks include @{ ... } blocks, as well as control structures. They’re Razor’s equivalent of <% ... %> blocks. The contents of a code block are emitted as-is in the Execute method. Code blocks must contain complete statements, or they’ll result in C# syntax errors.

VBHTML pages use @Code ... End Code blocks instead.

Code Nuggets are @-blocks. They’re Razor’s equivalent of <%: ... %> blocks in an ASPX page. Scott Guthrie describes how these blocks are tokenized. The contents of a code nugget are passed to the Write method, which is expected to HTML-escape its parameter and print it.

WebPages Side

The WebPages framework’s Write method (which comes from the WebPageBase class) takes a parameter of type Object, allowing one to put any expression in a code nugget. It passes its parameter to HttpUtility.HtmlEncode, which will call ToString() and HTML-escape the output. If the parameter is an IHtmlString, HtmlEncode will return its ToHtmlString() method without escaping.

The base class, method names, and default namespaces can be configured in the RazorEngineHost. In addition, custom RazorEngineHosts can override the PostProcessGeneratedCode method to make arbitrary modifications to the generated code.

The WebRazorHostFactory in System.Web.WebPages.Razor.dll can also read default namespaces, a default base type, and a custom host from the <system.web.webPages.razor> section in Web.config:

<system.web.webPages.razor>
    <host factoryType="MyProject.MyWebPageRazorHost" />
    <pages pageBaseType="MyProject.CustomWebPage">
        <namespaces>
            <add namespace="MyProject.SomeNamespace" />
        </namespaces>
    </pages>
</system.web.webPages.razor>

Next Time: Looking at the generated page

Dissecting Razor, part 3: Razor and MVC

2011-02-03T18:49:00+00:00

Last time, we saw how standalone Razor pages are served.

MVC3 maintains the strict separation between the WebPages framework and the Razor engine.1

Razor Side

Like the WebPages framework, MVC3 interacts with Razor indirectly, by relying on RazorBuildProvider from System.Web.WebPages.Razor.dll. However, MVC3 requires that Razor views inherit its own base class, System.Web.Mvc.WebViewPage.

MVC3 adds a new @model directive, which can be used instead of @inherits to specify a strongly-typed model. This syntax is implemented by customized RazorCodeParsers and RazorCodeLanguages in the System.Web.MVC.Razor namespaces. These classes are invoked by MvcRazorEngineHosts from a custom RazorHostFactory registered in Views/Web.Config:

<system.web.webPages.razor>
    <host factoryType="System.Web.Mvc.MvcWebRazorHostFactory, System.Web.Mvc, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" />
    <pages pageBaseType="System.Web.Mvc.WebViewPage">
        <namespaces>
            <add namespace="System.Web.Mvc" />
            <add namespace="System.Web.Mvc.Ajax" />
            <add namespace="System.Web.Mvc.Html" />
            <add namespace="System.Web.Routing" />
        </namespaces>
    </pages>
</system.web.webPages.razor>

MVC Side

On the MVC side, MVC3 includes a new RazorViewEngine which creates RazorView instances. RazorView inherits the existing BuildManagerCompiledView class, which passes the view’s virtual path to the build manager. RazorView will take the WebViewPage from the build manager, find any matching start pages, and execute the view.

As with the WebPages framework, one can substitute other templating engines. One can register a build provider which compiles classes that inherit WebViewPage, then add a RazorViewEngine to ViewEngines.Engines with additional extensions in its FileExtensions property.

Next time: Inside Razor Pages

Dissecting Razor, part 2: Gluing the pieces together

2011-02-02T16:46:00+00:00

Last time, we saw that ASP.Net Web Pages are implemented in two independent assemblies. These assemblies are not directly connected to each-other.

Razor Side

System.Web.WebPages.Razor.dll contains the RazorBuildProvider class, which allows ASP.Net’s build system to compile Razor pages. This class uses a WebRazorHostFactory to create WebPageRazorHosts used to process CSHTML (or VBHTML) files into CodeDOM trees. It compiles the CodeDOM tree and returns the generated type(which will typically inherit System.Web.WebPages.WebPage) to the build system.

WebPageRazorHost is coupled to the WebPages framework; it handles the non-standard base types for special pages (StartPage and ApplicationStartPage).

RazorBuildProvider can be configured to use a different WebRazorHostFactory that creates custom WebPageRazorHosts. (more on this later)

WebPages Side

The WebPages framework contains an internal WebPageHttpModule class which runs when an ASP.Net AppDomain is started. It runs any _AppStart files, and hooks the request lifecycle to handle requests for CSHTML (or VBHTML) pages.

Requests for Razor pages are handled by System.Web.WebPages.WebPageHttpHandler. This class passes the page’s virtual path to the build manager and gets a WebPage instance (It assumes that RazorBuildProvider will build the page and give a WebPage instance).

The handler calls the page’s ExecutePageHierarchy method to serve the page to the client. This method runs any _PageStart pages in the page’s parent directories, then executes the page.

The WebPageHttpHandler will also add a custom HTTP header, X-AspNetWebPages-Version: 1.0. This can be disabled by setting WebPageHttpHandler.DisableWebPagesResponseHeader to false.

As mentioned, System.Web.WebPages.dll is not directly tied to the Razor language and engine. One can create a custom build provider which compiles classes that inherit WebPage, then call WebPageHttpHandler.RegisterExtension to tell the WebPages framework to handle requests to the extension, without using the Razor parser.

The source code for this project is part of the same bundle.

Next time: MVC Razor views

Dissecting Razor, part 1: Parts of the framework

2011-02-01T03:51:00+00:00

Razor involves two distinct components: The Razor engine and the WebPages framework.

The Razor engine, in System.Web.Razor.dll, parses CSHTML (and VBHTML) files into CodeDOM trees. Except for the word Web in project name, the engine has nothing to do with ASP.Net; it doesn’t even reference System.Web.dll. In fact, it targets the .Net Client Profile, and only references mscorlib and System.dll.

The Razor engine is aware of all of Razor’s syntax-level features (code nuggets, sections, helpers), but is not aware of what they mean; it blindly transforms them into function calls.

The Razor engine can be used without ASP.Net for any kind of templating system. This is done by inheriting the RazorEngineHost class to provide default code-generation settings (such as base class and method name), then passing the host to a RazorTemplateParser.

The standard syntax will be annoying when generating non-XML-like content. To avoid this, one can write a custom MarkupParser to define a Razor syntax for a different markup language (such as CSS).

The WebPages framework, in System.Web.Webpages.dll, is a set of classes to use with the Razor parser. It contains the WebPage class which standard Razor pages inherit. This class defines the methods which are blindly called by the Razor parser, such as DefineSection and WriteLiteral. This framework also handles _PageStart and _AppStart pages and contains the HtmlHelper infrastructure.

The WebPages framework is not directly connected to the Razor parser. It could theoretically be used with a different template engine, as long as the template engine emits the correct method calls and class definitions. (more on this later)

The WebPages framework also contains two sets of utility methods. System.Web.Helpers.dll contains miscellaneous utility classes, such as Crypto and WebMail wrappers, plus grid and chart implementations. Microsoft.Web.Helpers.dll contains HTML helper classes which integrate with various third-party services, including Twitter, ReCaptcha, Google Analytics, and more. Most of these helpers can also be used in ordinary ASPX pages.

The source code for all of these projects is available here.

Next time: Gluing it all together.

Modifying HTML strings using jQuery

2011-01-31T13:55:00+00:00

jQuery makes it very easy to modify a DOM tree. For example, to strip all hyperlinks (<a> tags) from an element, we can write (demo)

$(...).find('a[href]')
      .replaceWith(function() { return this.childNodes });

After getting used to this, one might want to use jQuery to modify HTML contained in a string. Here, however, the naïve approach does not work:

var htmlSource = ...;
$(htmlSource).find('a[href]')
      .replaceWith(function() { return this.childNodes });

This code tries to remove all <a> tags from the HTML contained in the htmlSource string. However, what it actually does is create a detached DOM tree containing the new elements, strip all <a> tags in those elements, and throw the whole thing away. It doesn’t modify the original string. In fact, since the $ function only takes a reference to an immutable string, this approach cannot modify the original string.

Instead, you need to retrieve the source from the DOM tree after modifying it, then assign that source back to the variable.

There is an additional subtlety with this approach. jQuery cannot return the complete HTML source for a collection of elements. Therefore, it is also necessary to wrap the HTML in a dummy element (typically a <div>). One can then call .html() to get the innerHTML of the dummy element, which will contain exactly the desired content

This also eliminates the distinction between root-level elements and nested elements. If the original HTML string contains root-level <a> elements (which aren’t nested in other tags), writing $(htmlSource).find('a') won’t find them, since .find() only searches the descendants of the elements in the jQuery object. By wrapping the HTML in a dummy element, all of the elements in the original content become descendants, and can be returned by .find().

Here, therefore, is the correct way to modify an HTML string using jQuery:

var htmlSource = ...;
var tree = $("<div>" + htmlSource + "</div>");

tree.find('a[href]')
    .replaceWith(function() { return this.childNodes });

htmlSource = tree.html();

Generic Base Classes in ASP.Net MVC

2011-01-30T23:46:00+00:00

Last time, we saw that there are severe limitations in creating ASPX pages which inherit generic base classes. Many readers were probably wondering how ASP.Net MVC works around this limitation. In ASP.Net MVC views, people write pages like this all the time:

<%@ Page Language="C#" 
    Inherits="ViewPage<IEnumerable<DataLayer.Product>>" %>

ASP.Net MVC includes its own workaround for these limitations. The Web.config file in the Views folder of an ASP.Net MVC project registers a PageParserFilter:

<pages
    validateRequest="false"
    pageParserFilterType="System.Web.Mvc.ViewTypeParserFilter, System.Web.Mvc, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35"
    ...>
    ...
</pages>

PageParserFilter is one of ASP.Net’s lesser-known extensibility points. It can intercept different parts in the parsing process for an ASPX page and modify the page. The MVC framework’s ViewTypeParserFilter will check whether the page’s inherits="" attribute contains ( or a < characters; these characters can only appear in C# or VB.Net generic types, but not in the CLR’s native syntax.

If the inherits="" attribute contains these characters, it will save the attribute’s original value, then replace it with ViewPage (Or ViewMasterPage or ViewUserControl, as appropriate). This way, the rest of the built-in ASP.Net parser will see a normal type name that it knows how to parse. After the page finishes parsing, an internal ControlBuilder registered on MVC’s base types (ViewPage, ViewMasterPage or ViewUserControl) will replace the base type in the generated CodeDOM tree with the original value of the inherits="" attribute.

The one problem with the hack is that it leaves the ASPX parsing engine unaware of the page’s actual base type. Therefore, if you make a page that inherits a generic base class with additional properties, you won’t be able to set those properties in the <%@ Page declaration (since the ASPX parser won’t know about them). If you inherit a non-generic type, this mechanism will not kick in and page properties will work fine.

Generic Base Classes in ASP.Net

2011-01-27T16:55:00+00:00

ASP.Net pages can inherit from custom classes (as long as they inherit System.Web.UI.Page). This can be useful to add utility functions or shared (code-behind) behaviors to your pages. (Note that you could also use Extension methods or HTTP modules)

However, if you try to inherit a generic base class, it won’t work:

public class DataPage<T> : Page {
    public T Data { get; set; }
}

<%@ Page Language="C#" Inherits="DataPage<string>" %>

This code results in a yellow screen of death, with the parser error, Could not load type 'DataPage<string>'.

This happens because the ASP.Net page parser is unaware of C# generics syntax. The familiar generics syntax (eg, List<int>) is actually a C# innovation and is not used at all in the actual framework. The “native” generics syntax, which is used by Reflection, is markedly different: List`1[Int32] (namespaces omitted for brevity). This name is returned by the Type.AssemblyQualifiedName property.

Since ASP.Net uses reflection APIs to load types, we need to specify generic types using CLR syntax (and with full namespaces). Therefore, the following page will work:

<%@ Page Language="C#" 
    Inherits="TestSite.DataPage`1[[System.String, mscorlib]]" %>

However, it’s not so simple. ASP.Net does not call Type.GetType to parse these strings; instead, it loops over every referenced assembly and calls Assembly.GetType on each one. This is why you don’t need to include the assembly name whenever using the Inherits attribute (which would have been necessary for Type.GetType) Ordinarily, this is very useful, but here, it comes back to bite you.
It is not possible to parse a type from one assembly with a generic parameter from a different assembly using Assembly.GetType, unless the generic parameter is in mscorlib.

Therefore, for example, it is not possible to create an ASPX page that inherits DataPage<DataLayer.Product> if DataLayer.Product is in a different assembly than DataPage. As a workaround, one can create a non-generic class which inherits DataPage<DataLayer.Product>, then make the ASPX page inherit this temporary class.

Next time: MVC magic

Building Connection Strings in .Net

2011-01-26T14:20:00+00:00

.Net developers frequently need to build connection strings, especially when connecting to Access or Excel files using OleDB.
Code like the following has been written countless times:

//Bad code! Do not use!
string conn = "Data Source=" + openFileDialog1.FileName + "; "
            + "Provider=Microsoft.Jet.OLEDB.4.0;"
            + "Extended Properties=\"Excel 8.0\"";

This code looks innocuous at first glance, but will not work for all filenames. If the filename contains characters like ', ", ;, or =, this code will create an invalid connection string and throw an exception.

The correct way to build connection strings is to use one of the DbConnectionStringBuilder classes. This class implements a dictionary of key-value pairs in the connection string. It has a ConnectionString property which assembles the instance’s contents into a usable connection string. Unlike the string concatenation shown above, this property will correctly escape all values.

In addition, each of the database clients included with the .Net framework (SQL, OleDB, ODBC, Oracle, and Entity Framework) have their own inherited ConnectionStringBuilder classes in their respective namespaces. These classes add type-safe properties for the the keys supported by their databases, and handle any special cases when generating the connection string.

Thus, the correct way to write the above code is:

var connBuilder = new OleDbConnectionStringBuilder {
    DataSource = openFileDialog1.FileName,
    Provider = "Microsoft.Jet.OLEDB.4.0"
};
connBuilder["Extended Properties"] = "Excel 12.0 Macro";

As an added bonus, these classes implement ICustomTypeDescriptor, so they can be bound to a PropertyGrid to allow the end-user to edit the connection string. This can be seen in certain places in Visual Studio.

Don’t call Html.Encode in Razor Pages

2011-01-21T01:35:00+00:00

One of the unique features of ASP.Net WebPages (formerly Razor) is automatic HTML encoding. All strings printed by embedded code nuggets (@ blocks) are automatically HTML-encoded.

In addition to this feature, Razor also includes the Html.Encode method, probably copied from ASP.Net MVC. Calling this method naively leads to a nasty surprise – the string will be double-encoded!
To see why, look more closely at a typical call: @Html.Encode("<text>"). This Razor markup will call Html.Encode, which returns the string "<text>". Since it returns a string and not an IHtmlString, the Razor engine will encode it again, and render &lt;text&gt;.

Careful thought indicates that this behavior is probably correct. The programmer (hopefully) knows that Razor will escape its output, so the call to Html.Encode should be an attempt to display encoded text. In fact, this is the simplest way to display HTML-encoded text in a Razor view.

However, even if it is correct, the behavior is unexpected and should not be relied upon. The unambiguous way to display encoded text is to call Html.Raw:

@Html.Raw(Html.Encode(Html.Encode("Double-encoded <html> text!")))

Although it is long and clunky, this clearly shows that the text will be double-encoded.

Exercise for the reader: Why is it also necessary to call Html.Raw?

Optional Parameters in C# < 4

2011-01-19T22:04:00+00:00

C# 4.0 adds support for optional parameters.

The following code prints 4:

static void Main() {
    TestMethod();
}
static void TestMethod(int i = 4) {
    Console.WriteLine(i);
}

Optional parameters are a compiler feature. The compiler will emit a normal method with the IL [opt] attribute and a .param declaration that includes a default value:

.method hidebysig static void 
        TestMethod([opt] int32 i) cil managed
{
    .param [1] = int32(4)
    .maxstack 8
    L_0001: ldarg.0 
    L_0002: call void [mscorlib]System.Console::WriteLine(int32)
    L_0007: ret 
}

Earlier versions of the C# compiler will ignore this metadata. Therefore, you can call such methods in earlier versions of C#, but you will always need to pass the optional parameters.

You can also create methods with optional parameters in earlier versions of C#. You can force the compiler to emit this metadata using the [Optional] and [DefaultParameterValue] attributes.
(in the System.Runtime.InteropServices namespace)

The following C# 1 code will compile identically to the above:

static void TestMethod(
    [Optional, DefaultParameterValue(4)] int i
) {
    Console.WriteLine(i);
}

Since the older compilers cannot consume optional parameters, you will always need to pass i when calling this method from C# < 4. However, in C# 4, you can call this method without passing the parameter.

Unfortunately, the syntax parser used by VS2010 doesn’t recognize these attributes. Therefore, if you attempt to call a method using these attributes, inside the project that defines the method, without specifying the parameter, the IDE will show a syntax error. The compiler itself, however, will work correctly. I reported this bug on Connect.

The [DefaultParameterValue] attribute is optional; omitting it will use the type’s default value (0, an empty struct, or null, as appropriate) as the default.

These attributes can also be used to create methods with optional parameters using CodeDOM, which cannot emit the new syntax.

Writing output in Razor helpers using code

2011-01-12T20:22:00+00:00

The new ASP.Net WebPages view engine (formerly Razor) allows you to create reusable parameterized blocks of HTML called helpers.

For example:

@helper Fibonacci(int count) {
    int current = 1, prev = 0;
    for (int i = 0; i &lt; count; i++) {
        @:@current, 
        int t = current;
        current += prev;
        prev = t;
    }
}

This helper will write out the first count Fibonacci numbers. It can be called by writing @Fibonacci(30) in the page that defines the helper.

Using Razor syntax in this code looks strange. Razor syntax is designed to write HTML tags. Since I’m printing plain text, I need to use the @: escape (or the <text> tag) in order to output my text. This small bit of code looks confusing and can get lost inside larger blocks of server-side code.

Instead, I can use an undocumented hack. Razor helpers are implemented by compiling a lambda expression as an Action<TextWriter>. The lambda expression receives a TextWriter parameter named __razor_helper_writer. (You can see this by writing a Razor page with a compilation error and clicking Show Complete Compilation Source) There is nothing preventing me from using this parameter yourself. (it even shows up in IntelliSense!)

Therefore, I can rewrite the helper as follows:

@helper Fibonacci(int count) {
    int current = 1, prev = 0;
    for (int i = 0; i &lt; count; i++) {
        __razor_helper_writer.Write(current + &quot;, &quot;);
        int t = current;
        current += prev;
        prev = t;
    }
}

Remember to correctly escape your text by calling Html.Encode. Since this writes directly to the output stream, it doesn’t get the benefit of Razor’s automatic escaping.

Note: This relies on undocumented implementation details of the Razor compiler, and may change in future releases. I would not recommend doing this.
Instead, you can write a function:

@functions {
    string Fibonacci(int count) {
        var builder = new System.Text.StringBuilder();
        int current = 1, prev = 0;
        for (int i = 0; i &lt; count; i++) {
            builder.Append(current).Append(&quot;, &quot;);

            int t = current;
            current += prev;
            prev = t;
        }
        return builder.ToString();
    }
}

This can be called the same way as the helper. Instead of using the special helper support, the call will just print a string, the same you print any other string. Therefore, the function’s output will be HTML-escaped. To prevent that, you can change the function to return an HtmlString.

Binding to lists of DataRows

2011-01-10T20:30:00+00:00

.Net DataTables can be very useful when writing data-driven applications. However, they have one limitation: There is no obvious way to databind a grid (or other control) to an arbitrary list of datarows from a table.
You can bind to an entire table directly by setting a DataSource to the DataTable itself, and you can bind to a subset of a table by creating a DataView with a filter.

In general, you cannot bind to an IEnumerable<T> (eg, a LINQ query); the databinding infrastructure can only handle an IList (non-generic) or an IListSource. This is true for any kind of datasource. Therefore, to bind to any LINQ query, you need to call .ToList(). (Or .ToArray())

However, when binding to a DataTable, you can’t even use a List<DataRow>. If you try, you’ll get four columns (RowError, RowState, Table, and HasErrors) and no useful information. This happens because the List<DataRow> doesn’t tell the databinding infrastructure about the special properties of the DataRows. To understand the problem, some background is necessary

Databinding is controlled by the ListBindingHelper and TypeDescriptor classes. When you bind to a list, the ListBindingHelper.GetListItemProperties method is called to get the columns in the list. If the list implements the ITypedList interface, its GetItemProperties method is called. Otherwise, it will use TypeDescriptor to get the properties of the first item in the list. (this uses reflection)

The DataView class (which DataTable also binds through, using IListSource) implements ITypedList and returns DataColumnPropertyDescriptors that expose the columns in the table. This is why you can bind to a DataView or DataTable and see columns. However, when you bind to a List<DataRow>, there is no ITypedList that can return the columns as properties. It therefore falls back on reflection and shows the physical properties of the DataRow class.

To solve this issue, you need to wrap the list in a DataView so that you can take advantage of its ITypedList implementation. You can do that using the AsDataView() method. This method is only available on the DataTable and EnumerableRowCollection<T> classes; it cannot be called on an arbitrary LINQ query. You can only get an EnumerableRowCollection<T> by calling special versions of the Cast, OrderBy, Where, and Select methods from a DataTable.

Therefore, you can databind to a simple LINQ query by calling AsDataView() on the query. To bind to a List<DataRow>, or to a more complicated query, you can use an ugly hack:

List<DataRow> list = ...;
grid.DataSource = table.AsEnumerable()
                       .Where(list.Contains)
                       .AsDataView();

The AsEnumerable() call is not needed for typed datasets.

You can also call CopyToDataTable(), which will works on an arbitrary IEnumerable<DataRow>. However, it makes deep copies of the rows, so it isn’t helpful if you want the user to update the data, or if you want the user to see changes made (in code) to the original datarows.

Partial Type Inference in .Net

2010-12-30T01:58:00+00:00

When designing fluent APIs, one issue that comes up is partial type inference. If a method has two type parameters, there is no way to call it and only specify one of the type parameters (and leave the other inferred by the compiler)

For example, suppose we are creating a type-safe wrapper around a parameterized SqlCommand.
Ideally, it would be called like this:

using(DbConnection connection = ...) {
    var result = connection.ExecuteScalar<int>(
        "SELECT COUNT(*) FROM TableName WHERE Modified > someDate",
        new { someDate }
    );
}

Where the generic parameter specifies the return type.

In order to implement this efficiently, one would create methods at runtime which add DbParameters for each property in the anonymous type, and store them in a static generic class.
It would look something like this:

static class Extensions {
    public static void AddParams<TParam>(this IDbCommand command,
                                         TParam parameters)
        where TParam : class {
        if (parameters != null)
            ParamAdders<TParam>.Adder(command, parameters);
    }
    static class ParamAdders<TParam> where TParam : class {
        public delegate void ParamAdder(IDbCommand command,
                                        TParam parameters);
        public static readonly ParamAdder Adder = CreateParamAdder();

        private static ParamAdder CreateParamAdder() {
            return ...;
        }
    }
}

However, that requires that the ExecuteScalar extension method take TParam as a generic parameter. Since anonymous types can only be passed as generic parameters via type inference, this makes it impossible to pass the return type as a generic parameter.

To fix this issue, we can split the generic parameters across two methods. We can change the extension method to take a single generic parameter, and return a generic class with a method that takes the other generic parameter.

For example:

static class BetterExtensions {
    public static SqlStatement<T> Sql<T>(this IDbConnection conn,
                                         string sqlText) {
        return new SqlStatement<T>(conn, sqlText);
    }
}
class SqlStatement<TReturn> : IHideObjectMembers {
    public SqlStatement(IDbConnection connection, string sql) {
        Connection = connection;
        Sql = sql;
    }
    public IDbConnection Connection { get; private set; }
    public string Sql { get; private set; }

    public TReturn Execute() { return Execute<object>(null); }
    public TReturn Execute<TParam>(TParam parameters)
         where TParam : class {
        return ...;
    }
}

I use an IHideObjectMembers interface to hide the methods inherited from Object from IntelliSense. Note that the interface must be defined in an assembly outside of your solution. (or, to be more precise, that isn’t a Project reference)

This version would be called like this:

using(IDbConnection connection = null) {
    var result = connection.Sql<int>(
        "SELECT COUNT(*) FROM TableName WHERE Modified > someDate"
    ).Execute(new { someDate });
}

The return type is specified explicitly in the call to Sql<T>(), and the parameter type is passed implicitly to Execute<TParam>().

Simplifying Value Comparison Semantics

2010-12-29T03:10:00+00:00

A common chore in developing real-world C# applications is implementing value semantics for equality. This involves implementing IEquatable<T>, overriding Equals() and GetHashCode(), and overloading the == and != operators.

Implementing these methods is a time-consuming and repetitive task, and is easy to get wrong, especially GetHashCode(). In particular, the best way implement GetHashCode() is much more complicated than return x.GetHashCode() ^ y.GetHashCode().

To simplify this task, I created a ValueComparer class:

///<summary>
/// Contains all of the properties of a class that 
/// are used to provide value semantics.
///</summary>
///<remarks>
/// You can create a static readonly ValueComparer for your class,
/// then call into it from Equals, GetHashCode, and CompareTo.
///</remarks>
class ValueComparer<T> : IComparer<T>, IEqualityComparer<T> {
    public ValueComparer(params Func<T, object>[] props) {
        Properties = new ReadOnlyCollection<Func<T, object>>(props);
    }

    public ReadOnlyCollection<Func<T, object>> Properties
            { get; private set; }

    public bool Equals(T x, T y) {
        if (ReferenceEquals(x, y)) return true;
        if (x == null || y == null) return false;
        //Object.Equals handles strings and nulls correctly
        return Properties.All(f => Equals(f(x), f(y)));    
    }

    //https://stackoverflow.com/questions/263400/263416#263416
    public int GetHashCode(T obj) {
        if (obj == null) return -42;
        unchecked {
            int hash = 17;
            foreach (var prop in Properties) {
                object value = prop(obj);
                if (value == null)
                    hash = hash * 23 - 1;
                else
                    hash = hash * 23 + value.GetHashCode();
            }
            return hash;
        }
    }

    public int Compare(T x, T y) {
        foreach (var prop in Properties) {
            //The properties can be any type including null.
            var comp = Comparer.DefaultInvariant
                .Compare(prop(x), prop(y));    
            if (comp != 0)
                return comp;
        }
        return 0;
    }
}

This class implements an external comparer that compares two instances by an ordered list of properties.

ValueComparer can be used as a standalone IComparer<T> or IEqualityComparer<T> implementation.

It can also be used to implement value semantics within a type.
For example:

class Person : IComparable<Person>, IEquatable<Person>, IComparable {
    public string FirstName { get; set; }
    public string LastName { get; set; }
    public string Address { get; set; }
    public string Phone { get; set; }
    public string Email { get; set; }

    public override int GetHashCode() { return Comparer.GetHashCode(this); }
    public int CompareTo(Person obj) { return Comparer.Compare(this, obj); }
    int IComparable.CompareTo(object obj) { return CompareTo(obj as Person); }
    public bool Equals(Person obj) { return Comparer.Equals(this, obj); }
    public override bool Equals(object obj) { return Equals(obj as Person); }
    static readonly ValueComparer<Person> Comparer = new ValueComparer<Person>(
        o => o.LastName,
        o => o.FirstName,
        o => o.Address,
        o => o.Phone,
        o => o.Email
    );
}

To simplify this task, I created a code snippet:

<?xml version="1.0" encoding="utf-8" ?>
<CodeSnippets  xmlns="http://schemas.microsoft.com/VisualStudio/2005/CodeSnippet">
    <CodeSnippet Format="1.0.0">
        <Header>
            <Title>ValueComparer</Title>
            <Shortcut>vc</Shortcut>
            <Description>Code snippet for equality methods using ValueComparer</Description>
            <Author>SLaks</Author>
            <SnippetTypes>
                <SnippetType>Expansion</SnippetType>
            </SnippetTypes>
        </Header>
        <Snippet>
            <Declarations>
                <Literal Editable="false">
                    <ID>classname</ID>
                    <ToolTip>Class name</ToolTip>
                    <Default>ClassNamePlaceholder</Default>
                    <Function>ClassName()</Function>
                </Literal>
            </Declarations>
            <Code Language="csharp">
                <![CDATA[public override int GetHashCode() { return Comparer.GetHashCode(this); }
        public int CompareTo($classname$ obj) { return Comparer.Compare(this, obj); }
        int IComparable.CompareTo(object obj) { return CompareTo(obj as $classname$); }
        public bool Equals($classname$ obj) { return Comparer.Equals(this, obj); }
        public override bool Equals(object obj) { return Equals(obj as $classname$); }
        static readonly ValueComparer<$classname$> Comparer = new ValueComparer<$classname$>(
            o => o.$end$
        );]]>
            </Code>
        </Snippet>
    </CodeSnippet>
</CodeSnippets>

It can also be downloaded here; save it to My Documents\Visual Studio 2010\Code Snippets\Visual C#\My Code Snippets\

When shouldn’t you write ref this?

2010-12-22T22:54:00+00:00

Last time, we saw that the this parameter to an instance method in a struct is passed by reference, allowing the method to re-assign this or pass it as a ref parameter.

Due to limitations in the CLR, the this parameter to an iterator method is not a reference to the caller’s struct, and is instead a copy of the value. Quoting the spec (§7.6.7)

When this is used in a primary-expression within an instance method or instance accessor of a struct, it is classified as a variable. The type of the variable is the instance type (§10.3.1) of the struct within which the usage occurs.

If the method or accessor is not an iterator (§10.14), the this variable represents the struct for which the method or accessor was invoked, and behaves exactly the same as a ref parameter of the struct type.

If the method or accessor is an iterator, the this variable represents a copy of the struct for which the method or accessor was invoked, and behaves exactly the same as a value parameter of the struct type.

To explain why, you’ll need to understand how iterators are compiled. As Jon Skeet explains, an iterator method is compiled into a nested class that implements IEnumerable, with the original code transformed into a state machine. This nested class has fields to store the method’s parameters (which includes this for instance methods) so that the iterator code can use them.

This is why iterators cannot take ref parameters; the CLR cannot store a reference (as opposed to a reference type) in a field. Therefore, the this parameter is passed to iterator methods by value, not by reference.
This is also why anonymous methods in structs cannot use this; anonymous methods are also compiled to methods in separate classes and so cannot inherit the ref parameter.

This means that if you change the Mutate method from before into an iterator, the code will still compile (!), but the calling method will not see the changes.

static void Main() {
    Mutable m = new Mutable();
    m.MutateWrong().ToArray();    //Force the iterator to execute
    Console.WriteLine("In Main(): " + m.Value);
}

struct Mutable {
    public int Value;
    
    public IEnumerable<int> MutateWrong() {
        this = new Mutable();
        MutateStruct(ref this); 
        Console.WriteLine("Inside MutateWrong(): " + Value);
        yield break;
    }
}
static void MutateStruct(ref Mutable m) { 
    m.Value++; 
}

This code prints

Inside MutateWrong(): 1
In Main(): 0

In summary, don’t mutate structs in iterators (or at all, if you can help it).
I don’t know why this isn’t a compiler error.

When can you write ref this?

2010-12-22T20:50:00+00:00

Usually, you cannot pass ref this as a parameter, since this is not a writable field. However, that’s not true for value types. The this field of a value type is a writable value.

To quote the spec (§5.1.5)

Within an instance method or instance accessor of a struct type, the this keyword behaves exactly as a reference parameter of the struct type (§7.6.7).

Therefore, the following code prints 1:

static void Main() {
    Mutable m = new Mutable();
    m.Mutate();
    Console.WriteLine(m.Value);
}

struct Mutable {
    public int Value;
    
    public void Mutate() {
        this = new Mutable(); 
        MutateStruct(ref this); 
    }
}
static void MutateStruct(ref Mutable m) { 
    m.Value++; 
}

In practice, this should never come up, since mutable structs are evil and should be avoided at all costs.

Next time: This doesn’t always work.

Nothing vs Null

2010-12-22T17:28:00+00:00

VB.Net’s Nothing keyword is is not the same as C#’s null. MSDN states, “Assigning Nothing to a variable sets it to the default value for its declared type. If that type contains variable members, they are all set to their default value”.

In other words, the Nothing keyword is actually equivalent to C#’s default(T) keyword, where T is the type that the expression is used as.

This can lead to nasty surprises with nullable types in conditional operators.
In C#, the expression (...) ? null : 1 will not compile, since “there is no implicit conversion between '<null>' and 'int'”. Since null is an untyped expression, the type of the conditional is inferred to be int, resulting in an error because null cannot be converted to int.

In VB.Net, by contrast, the equivalent expression, If((...), Nothing, 1), will compile, but will have unexpected results. Here too, Nothing is an untyped expression, so the type of the conditional is inferred to be Integer. However, unlike null, Nothing can be converted to Integer, so this is compiled as If((...), 0, 1), which is probably not what the programmer intended.

In both languages, the solution is to use an expression which is actually typed as int?, by writing new int?(), (in C#) or New Integer?() (in VB.Net).

Animating Table Rows with jQuery

2010-12-21T17:19:00+00:00

jQuery contains a powerful and flexible animation engine. However, it has some limitations, primarily due to underlying limitations of CSS-based layout

For example, there is no simple way to slideUp() a table row (<tr> element). The slideUp animation will animate the element’s height to zero. However, a table row is always tall enough to show its elements, so the animation cannot actually shrink the element.

To work around this, we can wrap the contents of each cell in a <div> element, then slideUp() the <div> elements. Doing this in the HTML would create ugly and non-semantic markup, so we can do it in jQuery instead.

For example: Demo

$('tr')
    .children('td, th')
    .animate({ padding: 0 })
    .wrapInner('<div />')
    .children()
    .slideUp(function() { $(this).closest('tr').remove(); });

Explanation:

Get all of the cells in the row
Animate away any padding in the cells
Wrap all of the contents of each cell in one <div> element for each cell (calling wrapInner())
Select the new <div> elements
Slide up the <div>s, and remove the rows when finished.

If you don’t remove the rows, their borders will still be visible. Therefore, if you want the rows to stay after the animation, call hide() instead of remove().

Requiring Inherited Types in Generic Constraints

2010-12-17T17:08:00+00:00

A generic class can specify that its generic parameter must inherit a type. However, there is no obvious way in general to prevent clients from passing the base type itself.

For example, take the following set of types:

abstract class Entity  { }

class Person : Entity { }
class Boat : Entity { }
class Car : Entity { }

class Repository<TEntity> where TEntity : Entity { }

This allows the type Repository<Entity>, which doesn’t make logical sense.

In this particular case, we could prevent that by changing the generic constraint to where TEntity : Entity, new(). Since the base Entity class is abstract, that would disallow a Repository<Entity>. However,if the concrete entities also don’t have default constructors, this wouldn’t work. Similarly, had the base type been an interface, we could add a : class constraint.

There is an (somewhat) ugly hack that can be used to prevent parameterizations of the base class in arbitrary cases (as long as you control every type involved):

abstract class Entity  { }
interface IConcreteEntity { }    //Marker interface

class Person : Entity, IConcreteEntity { }
class Boat : Entity, IConcreteEntity { }
class Car : Entity, IConcreteEntity { }

class Repository<TEntity> 
      where TEntity : Entity, IConcreteEntity { }

Specifically, we can add a marker interface which is implemented by all concrete implementations of the base type. We can then constrain a generic parameter to inherit both the base type and the marker interface. Since the base class itself does implement the marker interface, it will not be valid as a parameter.
Note that the marker interface must be implemented (perhaps indirectly) by every single concrete implementation.

Nested Iterators, part 2

2010-12-16T21:50:00+00:00

In part 1, we discussed the simple approach to making a nested iterator. However, we fell short of a completely lazy nested iterator.

In simple cases, we can make an separate iterator method for the subsequence:

IEnumerable<IEnumerable<int>> FullyLazy() {
    for(int i = 0; i < 10; i++) 
        yield return Inner(i);
}
IEnumerable<int> Inner(int i) {
    for(int j = 0; j < 10; j++)
        yield return i * 10 + j;
}

Note that this is actually smaller than the single-method implementation!

This seems to work very well; the inner iterator code for a particular subsequence will not execute at all unless that subsequence is actually enumerated.

However, this approach falls short in practice. To see why, consider a real-world example.
Here is a fully lazy implementation of a Partition method, which converts a sequence into a 2D “jagged IEnumerable” where each subsequence (partition) contains n elements.

class Ref<T> { 
    public Ref(T value) { Value = value; } 
    public T Value { get; set; } 
}

public static IEnumerable<IEnumerable<T>> Partition<T>
              (this IEnumerable<T> sequence, int size) {
    using(var enumerator = sequence.GetEnumerator()) {
        var isFinished = new Ref<bool>(false);
        while(!isFinished.Value) {
            yield return PartitionInner(
                 enumerator, size, isFinished
            );
        }
    }
}
static IEnumerable<T> PartitionInner<T>
      (IEnumerator<T> enumerator, int size, Ref<bool> isFinished) {
    while(size --> 0) {
          isFinished.Value = !enumerator.MoveNext();
        if (isFinished.Value) yield break;
        yield return enumerator.Current;
    }
}

This method is complicated by the need to communicate back from the inner iterator to the outer one. Since iterators cannot have ref parameters, I need to make a “box” class that holds a reference to an int. This allows the outer iterator to find out when the sequence finishes.

In addition, this implementation has a subtle bug: If the last partition is full, it will return an extra, empty, partition after it.
Fixing this issue would require that the outer method call MoveNext() before each call to the inner method. This makes the code even more complicated; I won’t list it here (unless people really want me to)

This design has a more fundamental problem: The behavior of the outer method is determined by the inner ones. It will only work if each inner IEnumerable<T> is enumerated exactly once, before the next MoveNext() call to the outer iterator. If, for example, you iterate each inner iterator twice, it will behave unexpectedly:

Enumerable.Range(1, 8)
          .Partition(3)
          .Select(p => String.Join(", ", p.Concat(p)));

This code is intended to create strings that contain each partition twice. It is supposed to return

1, 2, 3, 1, 2, 3
4, 5, 6, 4, 5, 6
7, 8, 7, 8

However, it actually creates the strings

1, 2, 3, 4, 5, 6
7, 8

Enumerating the inner iterator a second time will end up consuming extra items from the original sequence.

Thus, in order to produce enumerables that behave correctly, the inner enumerator needs to cache its items; it is impossible to make a fully lazy Partition method.

It gets worse. In order to allow callers to write thingy.Partition(5).Skip(2), (which should skip the first two partitions) the outer enumerator needs to cache all items, because it cannot assume that the inner iterators will be called at all.

Thus, the laziest possible Partition method must use the original approach:

public static IEnumerable<IEnumerable<T>> PartitionCorrect<T>
              (this IEnumerable<T> sequence, int size) {
    List<T> partition = new List<T>();
    foreach(var item in sequence) {
        partition.Add(item);
        if (partition.Count == size) {
            yield return partition;
            partition = new List<T>();
        }
    }
    if (partition.Count > 0)
        yield return partition;
}

It would be possible to write a slightly lazier version that combines these two approaches and passes a List<T> to the inner iterator, which would return whatever is in the list, then enumerate if necessary to fill the list. However, I’m too lazy to do it. (unless people really want me to)

Nested Iterators, part 1

2010-12-16T16:26:00+00:00

C# 2.0 introduced a powerful feature called an iterator, a method which returns an IEnumerable<T> or IEnumerator<T> using the new yield keyword.

Using an iterator, you can quickly and easily create a method which returns lazily a sequence of values. However, lazily returning a sequence of sequences (IEnumerable<IEnumerable<T>>) is not so simple.

The obvious approach is to yield return a List<T>:

IEnumerable<IEnumerable<int>> SemiLazy() {
    for(int i = 0; i < 10; i++) {
        List<int> numbers = new List<int>();
        for(int j = 0; j < 10; j++)
            numbers.Add(i * 10 + j);
            
        yield return numbers;
    }
}

(This can be shortened to a single LINQ statement, but that’s beyond the point of this post: Enumerable.Range(0, 10).Select(i => Enumerable.Range(10 * i, 10)) )

This approach is very simple, but isn’t very lazy; each subsequence will be computed in its entirety, whether it’s consumed or not.

This approach also has a subtle catch: the iterator must return a different List<T> instance every time.

If you “optimize” it to re-use the instance, you’ll break callers which don’t use the subsequences immediately:

IEnumerable<IEnumerable<int>> Wrong() {
    List<int> numbers = new List<int>();
    for(int i = 0; i < 10; i++) {
        numbers.Clear();
        for(int j = 0; j < 10; j++)
            numbers.Add(i * 10 + j);
            
        yield return numbers;
    }
}

Calling SemiLazy().ToArray()[0].First() will return 0 (the first element in the first subsequence); calling Wrong().ToArray()[0].First() will return 90 (since all subsequences refer to the same instance).

Next: How can we achieve full laziness?

On copy prevention in HTML, part 3

2007-04-27T20:56:00+00:00

My previous post stretched the limit of simple copy prevention. Beyond this point, it gets very complicated. Before continuing, some thought is in order. Who are you trying to prevent from copying your text? Why shouldn't the text be copied? Unless you are trying to stop a hardcore developer, the previous methods should suffice. Also, what kind of copying are you trying to prevent? If you are trying to prevent the copier from copying into a web page, it is significantly harder, because he can copy your source and it will display normally.

I can think of two ways to prevent the copier from using a screenreader to copy your text.

Put all of the text into a single, CAPTCHA-like image. This way, the screenreader will not be able to read the text. However, this will also make it more difficult for legitimate people to read your text. Also, the copier could simply insert the large image as-is into his document. This risk could be mitigated by watermarking it with your name.
Break apart the text into many different images, each one somewhat smaller than a letter. This could be done in server-side code. Then, use a JavaScript timer to alternate the images so that the screenreader will never see all of the text at once. To prevent the copier from modifying the JavaScript to show all of the images, alternate them with other images. For example, write a server-side script that takes X and Y coordinates, and a timer index. This script would either a white image, or the chunk of text at the given coordinates, depending on the timer index. To prevent the copier from using GDI to OR-blit screenshots from different times, (or from using Photoshop to make the white transparent, then pasting them together) the white could have random black patterns. However, this will make the text hard to read. If a JavaScript timer isn't fast enough to make the text legible, it could be done in Flash.

Preventing the copier from copying your content into HTML is more difficult. No matter how obscure your source is, the copier could use a tool like Firebug to copy your DOM source using the innerHTML property. Here too, there are several options.

Use my Scrambler (see part 2), and put your name anywhere within the scrambled text. (for example, you could put in, in the middle, Written by Your Name Here; do not copy) It is virtually impossible for the copier to extricate the spans that form your name and then fill the resulting gap. This could also be done in an image. However, if you put your name at the end, the copier could position a white DIV to hide it. Or, if his name is similar in length, he could position a white DIV with his name to hide it.
If you are only worried about part of your text, scramble the entire page. It would be extremely difficult for the copier to extract the sensitive part. For example, you could add a lengthy copyright header before the content, and scramble it with the content. However, the copier could position the containing DIV so that the copyright header is above the top of the DIV.
Break the text into a large set of images, and use client-side JavaScript to execute a server-generated script that adds these images from another server-side script. For every request, require a single-use authorization token returned by the previous request. The initial page request would include the auth-token for the first script, and each image would be preceded by an AJAX request for auth-tokens for the image and for the next AJAX request. The first script would have an auth-token for the first AJAX request embedded within it. To make it more difficult for an attacker to get an auth-token from the initial script, you could encode the script before sending it, and decode on the client, then pass it to the eval() function. All responses would have the no-cache header to prevent the copier from taking the images out of the cache, and the images would be used by the background-image attribute on a DIV to prevent Save Image As (only necessary if Save Image As doesn't redownload the image from the server). The server would track auth-tokens in a database, and delete them when used. If you do all this, the only way the copier could get the images from the page would be Print Screen. To prevent that, make the images flicker as described earlier. The copier could, however, load your page with JavaScript disabled, download and decode the script, convert it to a full programming language (eg, C#), and use the script's embedded auth-token to send the "AJAX" requests over HTTP (for example, using .NET's HttpWebRequest class) and download the images. To prevent this, make the auth-tokens expire after about 30 seconds. If any auth-token is expired, all subsequent images should form something different. (maybe Service Unavailable, or random black pixels, or a different text) By the time the copier finishes writing his program, his "stolen" auth-token will have expired, and he will not know what he did wrong. To prevent him from trying again, you could blacklist his IP address after receiving an expired request, and embed IP address in auth-tokens. You could also set and require some innocuous-seeing cookies on the server for every request. The copier wouldn't notice these cookies, and when he requests the images without these cookies in his request, you could send him whatever you want. Please note that if your page requires a login, the copier probably will check cookies. And, if the copier is being paid by the hour (this is quite likely; otherwise, he'd give up), he might even thank you for doing all this.

On copy prevention in HTML, part 2

2007-04-16T21:49:00+00:00

The methods discussed in my previous post are crude and ugly. Most of the time, they do work, but they do nothing to prevent the user from viewing the source and copying the text from there. Also, the user has a right to select text that should not be denied. For example, if one wants to show someone part of a large document, the easiest way to do that is to select the part.

ZSkTuKpBrLljyVW GmtoBbO MVocxRvoopy zKYtQahiDEsh LLtQexowSEtDnIg. NoyticDMe thiMaDVnZt, whGZenjEE ufapdeIPasBZxtgCeYWDd, iSt BlMNooks KPzRlkeeGifkshqdheodB tIVnoMNtEal nySouQnAqVsensegX. cUHHoNweqdcvecFGrU,PGZ pMibt rqcanrbKkn eHstilqTulOE beRPuv STwaQsyTePXvplRoCectxeKAjd jVpnXljoDYrDlrmaKlly.B IiLwokzofk at itsjw JXsCoIuuFhjrce:

<SPAN style="position: static;left:-9477px;position: absolute;">Z</SPAN>
<SPAN style="position: static;left:-9765px;position: absolute;">S</SPAN>
<SPAN style="position: static;left:-9586px;position: absolute;">k</SPAN>
<SPAN style="position: static;left:-9373px;position: absolutee;">T</SPAN>
<SPAN style="position: static;left:-9734px;position: absolute;">u</SPAN>
<SPAN style="position: static;left:-9872px;position: absolute;">K</SPAN>
<SPAN style="position: static;left:-9773px;position: absolute;">p</SPAN>
<SPAN style="position: static;left:-9326px;position: absolute;">B</SPAN>
<SPAN style="position: static;left:-9195px;position: absolutee;">r</SPAN>
<SPAN style="position: static;left:-9413px;position: absolute;">L</SPAN>
<SPAN style="position: static;left:-9196px;position: absolute;">l</SPAN>
<SPAN style="position: static;left:-9737px;position: absolute;">j</SPAN>
<SPAN style="position: static;left:-9897px;position: absolutee;">y</SPAN>
<SPAN style="position: static;left:-9014px;position: absolute;">V</SPAN>
<SPAN style="position: static;left:-9893px;position: absolute;">W</SPAN>
<SPAN style="position: static;left:-9103px;position: absolutee;"> </SPAN>
...

gicotxaoyenc:yehosrk,mutaryslts.tsitatkiisuoptcrebhttnihdoLfI

<span style="position: absolute; left: 241px; top: 0px;">g</span>
<span style="position: absolute; left: 263px; top: 0px;">i</span>
<span style="position: absolute; left: 117px; top: 0px;">c</span>
<span style="position: absolute; left: 17px; top: 20px;">o</span>
<span style="position: absolute; left: 276px; top: 0px;">t</span>
<span style="position: absolute; left: 287px; top: 0px;">x</span>
<span style="position: absolute; left: 129px; top: 0px;">a</span>
<span style="position: absolute; left: 9px; top: 20px;">o</span>
<span style="position: absolute; left: 13px; top: 0px;">y</span>
...

These texts were generated by a script (currently unavailable). It can do two things: Inflate (the first paragraph above) and Scramble (the second paragraph).

Using the Inflate option will add random characters in SPANs positioned absolutely between nine and ten thousand pixels to the left. When the user selects text, the random characters will also be selected, and when the text is copied, they will also be copied. It will randomly add up to five letters between every two characters, and it will ignore text in TEXTAREAs, SCRIPTs, and SELECTs. Such text could be "deflated" by copying the source, then removing all text that matches the regular expression /<span[a-z0-9 ;:="']*?>[A-Za-z]</span>/ , removing all SPAN tags that contain a letter and an attribute. Therefore, I put each original letter in a very similar SPAN tag, complete with a random location, and gave both types of spans position:absolute and position:static, in different orders. This could also be matched by a regular expression, but it would be much more complicated, and I will not list it here. It would also be possible to write a GreaseMonkey script that would loop through the SPANs and delete all of them which has a position attribute equal to absolute. However, it would probably be easier to retype it manually.

Using the Scramble option will put each letter in to a SPAN, position them absolutely at their correct location, and randomize the order. Therefore, when the user selects the text, the selection will be scrambled, and when it is pasted, it will show up as nonsense. Spaces are rendered pointless by the procedure, and are therefore removed. Scrambled text will not select cleanly, but is nearly impossible to descramble. It would be possible to write a GreaseMonkey script that would sort the SPANs by top, then by left, but it would be much easier to retype the text manually.

When using this approach, remember to position the text's container, or it will show up in unexpected places. In addition, this approach will completely break word wrap,and must therefore be placed in a container with a fixed width. This width should be entered in the Width textbox in the scrambler so that the text will flow correctly.

These methods will definitely prevent all but the most determined and technically skilled copiers. However, they do not prevent OCR screenreaders. This will be discussed in part 3.

On copy prevention in HTML, part 1

2007-04-08T18:27:00+00:00

Many web developers like to prevent their viewers from copying their text. While I do not approve of this, there are cases where it is appropriate.

The simplest way to achieve this is to use the IE only attribute UNSELECTABLE and the FireFox only css style -moz-user-select. Such HTML looks like this:

<DIV unselectable="on"
style="-moz-user-select:none;">
You can't select me.
</DIV>

You can't select me.

To make the HTML and CSS validate, one could do this in Javascript: Elem.unselectable = "on"; Elem.style.MozUserSelect = "none";

However, this method only works in IE and Firefox. In addition, in IE, it doesn't work very well, and if a user tries hard, he will end up selecting the text.

A slightly better way to do it is to handle the onselectstart event (for IE) and the onmousedown event (for everything else) and return false. This will prevent the browser from handling the events. This results in something like this:

<DIV
onselectstart="return false;"
onmousedown="return false;" >
You can't select me.
</DIV>

You can't select me.

The problem with these methods is that they do nothing to prevent a user from reading the HTML source. This is discussed in the next part.

SLaks.Blog

Code Snippets: Removable = Tokens

1: The Color Color trick

Variant: Casting

2: Collapsing 2 tokens to 1.

3. Invocation expression

Variant: ref return

4. Lambda expressions vs. Expression-bodied methods

5. Splitting a class member in half

Would-be variant:

6. Tuple Deconstruction

7. Local functions

Common Cryptographic Pitfalls

Don’t re-invent the car

Don’t re-invent the wheel

Learn as much as possible

Don’t reuse keys

Create a threat model

Use IVs

Don’t reuse IVs

Don’t expose error messages or other internal information

Validate everything

Chains of Trust – The Root of all Security

The Web Authentication Arms Race – A Tale of Two Security Experts

Theory vs. Practice

Authenticating the user

Trusting hardware

Forgot password

Handling margins in CSS

Code Snippets: Impossible Code

Impossible accessibility

void expressions

Impossible overload resolution

Generic constraint conflict

Impossible inheritance

Interface method conflict

Code Snippets: Fast Runtime Property Access with Reflection

Code Snippets: And in the darkness bind() them

Code Snippets: Variadic Generics in C#

Code Snippets: Conditional Casting

Concurrency, part 6: Easier asynchrony in C# with await

Non-blocking IO

Where do asynchronous operations come from?

So what are async methods good for?

Pitfalls

Concurrency, part 5: Advanced Promise Usage

Error handling

Calling asynchronous methods in loops

Parallel Operations

Sequential operations

Passing state along a promise chain

Caching asynchronous operations

Challenges in Silon

Visual Studio Code under the hood

Typename Comments, part 2: The method behind the madness

Introducing Typename Comments – A New Kind of Comment

Count() Considered Occasionally Harmful

Concurrency, part 4: Comparing promises frameworks in different languages

.Net languages (C#, VB, etc…)

Methods

Behavior

Availability

Java 7 – Guava

Methods

Java 8

Methods

Availability

Javascript

Methods

Availability

jQuery Promises

Methods

Availability

Concurrency, part 3: Promises – Asynchronous programming made easy

Basics

Creating promises

Composing promises

Error handling

Unit testing

Concurrency, part 2: Patterns for Asynchronous Methods

1: The `Color Color` trick

Variant: `ref` return