Self Hosting and http.sys

In the last week I've gone through a little nice "journey of discovery" where my current explorations with self hosted Web API and self hosted SignalR have led me to fully understand some stuff I've been partially aware of since 10 years ago (I've been putting together different small pieces until conforming a whole, very interesting, picture).

Back in 2006, while doing a short dive into Python's Web Development ecosystem (Django, Turbogears...), I felt delighted by the presence of basic http server modules that allowed you to test your applications sparing you from having to install and configure a "real" server and furthermore affording you to write applications with an embedded Web Server. Later on I found this idea again in other platforms like node.js or .Net WCF (that has featured self-hosting since its first version).

As of late, many other .Net libraries/frameworks provide self hosting functionality (Web API, SignalR via Owin, Service Stack...) this led me to think that most likely the .Net framework includes the core functionality for creating your own web server. Some searching took me to this StackOverflow question. So it seems like all the magic is provided by System.Net.HttpListener (it appears like SignalR directly uses HttpListener while WCF and Web API use an extra layer on top of it). Notice also that this has nothing to do with Hosted Web Core Applications that rely on IIS.

Well, HttpListener brings this embeddable Web Server to your application, but it does not implement the core http logic, this seems to be in the http.sys driver. I've been vaguely familiar with http.sys since the Window 2003 times. IIS6 meant an important evolution of our beloved Web Server, and the fundamental part came given by the introduction of http.sys, a kernel driver that, to put it simple, takes care of the http (and https) protocol. When you create a HttpListener, it registers itself with http.sys, so that http.sys routes traffic to your listener based on the url-prefix that you've registered. It's nicely explained in this article:

Basically, http.sys is a kernel-mode listener that has intrinsic knowledge of HTTP. Different parties can register with this listener to have requests forwarded to them. Typical examples nowadays include IIS 6 and SQL Server 2005

and in this other excellent one. There I've learnt that HttpListener is a managed wrapper around the HTTP Server API.

Notice that http.sys is a core component of the Windows networking subsystem, so it'll be present in any Windows PC from Windowx XP SP2 on. You can read more about it here, and you can verify that http.sys is installed in your system by typing: driverquery on your command line.

Opening listeners on http.sys is protected by ACL's, so unless the application creating the listener is running as Admin, you'll get an AddressAccessDeniedException. So, you'll have to use the netsh tool (running it as Admin) to add permissions for the user (that will later on run the application) to register a listener on the intended url-prefix. For example:
netsh http add urlacl url=http://+:80/MyUri user=DOMAIN\user

One of the coolest things with http.sys is that it allows port sharing between http applications, that is, different applications can be registered for the same port (but different url-prefix, I mean: app1 registers for localhost:80/app1 and app2 for localhost:80/app2). Obviously this is possible cause the applications do not listen directly on that port, it's http.sys who does it and then forwards accordingly to one or another application. This port sharing feature has been extended to other protocols by means of Net.TCP Port Sharing.

At the end of this excellent reading you'll find that http.sys is at the core of one of today's most confusing and painful topics on the modern web development arena, why on earth doesn't .Net 4.5 (and therefore WCF, SignalR...) support WebSocket in OS's other than Windows8 and Windows Server 2012?

The problem is with the way that HTTP.sys works in Windows 7/Server 2008/R2 and earlier OS versions. Rather than issue a patch that could result in large regressions for existing customers that rely on the current behavior of HTTP.sys, the newest versions of Windows include a modification to HTTP.sys that enables WebSockets.

This other post delves deeper into what "the way that http.sys works in Windows 7" means:

The current technical issue for our stack is that the low-level Windows HTTP driver that handles incoming HTTP request (http.sys) does not recognize the current websockets format as having a valid entity body. As you have noted, the lack of a content length header means that http.sys does not make the nonce bytes in the entity available to any upstack callers. That’s part of the work we will need to do to build websockets support into http.sys. Basically we need to tweak http.sys to recognize what is really a non-HTTP request, as an HTTP request.

I've read somewhere the WebSocket support in Windows 8/2012 is built in websocket.dll, so my guessing is that the "websockets enabled" http.sys in these OS's will take care of the "almost http like" websocket handshake, and then the normal communication will be handled by websocket.dll.

As Technology Evolves

Sometimes it's hard to realize how fast technology moves, yes, even for programming freaks like me. Truth is that probably I have fewer technological needs than others. First, I don't give a shit about facebook or about locating my (real) friends on a map in real time (and much less about being located by them). I don't need to publicize that I've just taken a pee or eaten an orange (like other people seem to desperately need), I've never liked IM programs, I consider them deeply intrusive and only use them at work, for normal life email seems much more gentle and appropriate to me. So, I don't pay for a data connection in my smartphone, I've got internet at home, at work and at almost any cafe in Xixón where I occasionally hang out. For me, a data connection in my phone would only be useful when abroad, but abroad you can't use it because of its exorbitant price (hopefully almost any cheap hostel today has free WiFi :-)

I'm a rather organized, methodical person when it comes to travelling (shame this does not apply to other aspects of my life), so apart from doing tons of previous "research" about the history and social issues of any place I plan to go, I ever take with me tons of maps, wikipedia articles and so on saved in my (cheap) smartphone and tablet. I've had one kind or another of smartphone since September 2007 (at that time we called it PDA + Phone + GPS), and believe it or not, it was not until this month that I've really needed to use my current device's GPS (I got a bit lost in Dresden when trying to reach my hostel from the Neustadt Bahnhof, allegedly a 5 minutes walk, but when you start your path right on the contrary direction because your "biological compass" tends to confuse north and south... the route can turn more complex...) My GPS helped me cause I had an offline google map of Dresden (offline google maps has been for me one of the breakthroughs in the last year, though for most people it has gone mainly unnoticed).

Well, what's prompted me to write this post is another technological advance that has been around for some time but I'd never used so far, Google Search by Image. Months ago I tried to find a harsh, moving image that I profoundly conveys the desolation brought by war. I'd seen it many times before, but I didn't know the author, I didn't have any copy of it (save for a "brain copy"), and for some reason I mistakenly thought it was related to the Crimean War (which was not possible cause at that time photography was at its early stage). So, all my attempts to locate the picture were to no avail. The other day, just by chance, I came across a blog that was using the image as header, but it had no extra information about it. Today, after providing the url for that picture to the Search by Image functionality, Google came out with this.

So this extraordinary picture is called (quite appropriately) "Grief" and was taken by Dmitri Baltermants. I was not that confused, cause the pic was taken in Crimea, but not during the Crimean War, but during the WWII (shit, there are places that have been devastated by war again and again). This page makes for a very interesting reading.

The photo depicts a 1942 Nazi massacre in the Crimean village of Kerch. Village women searched for the bodies of their loved ones. The contrast between the oversaturated sky above and the bodies haphazardly strewn in the foreground underlines the poignancy of the moment, but for the same reason, the photo was censored in the Soviet Union where authorities only published the photos that could help boost morale; ‘Grief’ reflected nothing but harsh tragedies of war, and it wasn’t seen by the general public until the 1960s.

Windows Impersonation, Access Tokens and so on

I think anyone that has done some kind of Asp.Net development should have a basic idea of what Windows Impersonation is (running a thread impersonated under a user different from the one running the process), but well, let's delve into this a bit more.

When a process is run by a user (runs under the security context of that user), it means that the Access Token of that user gets assigned to the Process, and will be used for any security checking done when the process tries to access a securable resource. This token is a Primary Token. As for the threads of a process, they can or can not have an associated Access Token. If this token is present, it means that such thread is using impersonation, otherwise it's not impersonating. If impersonating, this Token (Impersonation Token) is used to validate the interaction of the thread with a securable object, if not impersonating, the Process Token (Primary Token) is used for these checks. You can read a good explanation here

Access Token -An access token contains the security information for a logon session. The system creates an access token when a user logs on, and every process executed on behalf of the user has a copy of the token. The token identifies the user, the user's groups, and the user's privileges. The system uses the token to control access to securable objects and to control the ability of the user to perform various system-related operations on the local computer. There are two kinds of access token, primary and impersonation.

When a Thread is created it does not get associated any Access Token (Impersonation Token), so it will run in the Security Context of its Process. This is so irrespective of whether the parent thread is or is not being impersonated. Likewise, when a new Process is created, the "impersonation status" of the calling thread does not have any effect, the new Process will get a Primary Token like the one of the calling Process (unless that we're running it under a different user by means of CreateProcessAsUser, CreateProcessWithLogonW or CreateProcessWithTokenW. This is pretty well explained here

So note that threads are always created without any Impersonation Token (CreateThread and its .Net wrappers don't have any token or user-password parameters), it's later on, while running, when the Thread can impersonate itself (and undo such impersonation later on). The thread would first obtain a Primary Token, that would use to create an Impersonation Token.
.Net Base Library does not provide any managed equivalent, so we'll need to resort to PInvoke:

 public class NativeSecurityHelper
    {
        public const int LOGON32_LOGON_INTERACTIVE = 2;
        public const int LOGON32_LOGON_NETWORK = 3;
        public const int LOGON32_LOGON_BATCH = 4;
        public const int LOGON32_LOGON_SERVICE = 5;
        public const int LOGON32_LOGON_UNLOCK = 7;
        public const int LOGON32_LOGON_NETWORK_CLEARTEXT = 8;
        public const int LOGON32_LOGON_NEW_CREDENTIALS = 9;
        public const int LOGON32_PROVIDER_DEFAULT = 0;

        [DllImport("advapi32.dll", SetLastError = true)]
        public static extern bool LogonUser(
            string lpszUsername,
            string lpszDomain,
            string lpszPassword,
            int dwLogonType,
            int dwLogonProvider,
            out IntPtr phToken
            );

        //I'll use it mainly to create an Impersonation Token from a Primary Token
        [DllImport("advapi32.dll", SetLastError = true)]
        public static extern bool DuplicateToken(IntPtr ExistingTokenHandle,
            int SECURITY_IMPERSONATION_LEVEL,
            out IntPtr DuplicateTokenHandle);
}

and we would use it like this:

        public static WindowsImpersonationContext ImpersonateCurrentThread(string domain, string userName, string password)
        {
            IntPtr primaryToken;
            IntPtr impersonationToken;
            //if it's a local logon, LogonUser expects "." for the domain
            domain = domain ?? ".";
           
            NativeSecurityHelper.LogonUser(userName, domain, password,
                                NativeSecurityHelper.LOGON32_LOGON_INTERACTIVE, NativeSecurityHelper.LOGON32_PROVIDER_DEFAULT, out primaryToken);
                
            //I think LogonUser returns a Primary Token, we have to duplicate it to obtain an Impersonation Token
            NativeSecurityHelper.DuplicateToken(primaryToken, 2, out impersonationToken);

            return ImpersonateCurrentThread(impersonationToken);
        }

        public static WindowsImpersonationContext ImpersonateCurrentThread(IntPtr impersonationToken)
        {
            WindowsIdentity windowsIdentity = new WindowsIdentity(impersonationToken);
            WindowsImpersonationContext impersonationContext = windowsIdentity.Impersonate();
            return impersonationContext;
        }

On the other hand, the case of creating a Process under a different account (for which we have its user and password) is contemplated by the Base Library straightaway, we can do it like this:

var process = new Process();
// Configure the process using the StartInfo properties.
process.StartInfo.FileName = processPath;
process.StartInfo.WindowStyle = ProcessWindowStyle.Normal;

//this is specific to the RunAs functionality:
process.StartInfo.UseShellExecute = false;

process.StartInfo.UserName = "testUser";
string password = "axSt23uv";

SecureString secPass = new SecureString();
foreach (char c in password)
{
 secPass.AppendChar(c);
}
process.StartInfo.Password = secPass;
process.Start();

Yes, having to create a SecureString from a normal string by appending chars instead of directly accepting a string in a constructor seems rather odd to me too.

Notice that I've defined 2 impersonation methods, one that expects a username and a password, and another one that expects an Access Token. This is the norm in Win32 API, some functions expecting user and pass, and others expecting a Token. In .Net we can obtain the identity under which a thread is running (impersonated or not) with just this line:

System.Security.Principal.WindowsIdentity.GetCurrent()

and from that WindowsIdentity object we can get its associated token via the Token property. Unfortunately, it simply returns a handler (an IntPtr) to the token, so if you were to want to get or set fields in the token, you would have to use the native GetTokenInformation and SetTokenInformation functions.

Pasha 183 has died, RIP

It's been sad to read that the outstanding Moscow's street artist Pasha 183 has died, and even more sad knowing that he was just 29 years old. I wrote a short mention to him in this blog 1 year ago. His work is impressive, that much as the work of that other genius everyone seemed to liken him to, Banksy. It's obvious that both shared many things: a boundless creativity, an urge for criticizing all what they found wrong in this world, a need for trying to open others eyes, or maybe to show them that they're not alone.

There's another obvious trait that both share and about which I feel like reflecting on, Anonymity.
Yeah, it's clear that the average citizen does not know their real names or faces. I'm saying "average citizen" here cause at least in the case of Banksy (who has done tons of works out of his country and at known dates) it's ridiculous to think that the British Police does not know who he is. Apart from having done most of his works in a city where there's a security camera on every corner (cameras that could reconstruct his route from the place where he did his work with a mask and a hood up, to the far place where he put his mask...) the other essential point is having done so many works abroad. It's just a matter of crossing data, the dates when Banksy works appeared in Barcelona, Melbourne, Gaza... and the dates of entry and exit of British citizens in other countries. Provided those data, how much CPU power do you think it would take to find out his name and face? :-D

So, what do I find so important about their condition of Anonymous Heroes?. For me, in these times of massive and radical egocentriticy and narcissism, statements like this by Pasha 183:

What I do is more important than who I am.
People should know my work, not my face.

are truly revolutionary acts of dignity in a world of dishonesty. In a society where Social Shit like facebook or the misuse of valid tools like twitter or video streaming services have made most of the population lose awareness of what privacy means and of the difference between virtual "friends" and real friends. Have made them unable to grasp the distinction between what is important and what is purely foolish, and going further in that sense, has made them believe that any futile and stupid thing that they happen to do or that happens to occur in their happy miserable lives is worth to be told to everyone through their "walls", their tweets, their stupid videos... Sharing wisdom is an act of generosity that fosters progress, sharing stupidity is an act selfishness and irresponsibility that promotes foolishness and alienation.

It's curious that several media mention the "glasses on the snow" piece as one of his most notorious works, while it's one of the less appealing to me (it's really creative, but it fades when compared to other of his works, mainly those with a more direct sociopolitical inspiration). I specially like this one:

The selection of works found here makes for a pretty good introduction to his creative universe. RIP

Decorate With Defaults Checking/Setting

The other day I ended up with a design where I would have a bunch of JavaScript functions where all of them would have a number of optional parameters that if not provided should be set to some configuration defined defaults. So I would have to repeat the two first lines in the function below in all these related functions

function getValue(x, y, server, credentials){
   server = server || Config.server;
   credentials = credentials || Config.credentials
   ...
}

Repeating this inside every function adds quite unnecessary noise, so thinking of a way to improve it I ended up writing a generic decorateWithDefaultChecking function. Nothing too interesting, we have a factory function that returns a new function that takes care of the defaults checking and then invokes the original function, but anyway I thought I would share it here:

//decorates a given function with checking/assignment of default values to undefined parameters
function decorateWithDefaultsChecking(targetFunc /* ,default1, default2... */){
  var defaults = [].slice.call(arguments, 1)
 if (!defaults)
  return targetFunc;
 var opStart = targetFunc.length - defaults.length;
 return function(){
  for(i=0; i<defaults.length; i++){
   if (arguments[opStart + i] === undefined){
    arguments[opStart + i] = defaults[i];
   }
  }
  //the next line below here is essential, remember that arguments is not a real array, but an "array like" object, 
  //so in order for this to work we need to set the length property on our own, otherwise, the new assigned values are ignored when invoking apply
  arguments.length = arguments.length + defaults.length;
  return targetFunc.apply(this, arguments);
 };
}

The most interesting point is one more JavaScript quirk that I stumbled upon. I've already posted about the beautiful arguments variable available inside each function, and already mentioned before that it's not a real Array, but an Array-like object. Well, in this case, its Array-like nature means we need to set its length property accordingly. Otherwise, though the addition of new elements to the pseudo-array takes place, they won't be passed to the function invoked via Function.prototype.apply

.