At work we’ve made the jump from back-end coding in Visual Studio and running our APIs on Windows with IIS to coding and hosting everything on Linux. We also switched from SQL Server to Postgresql. It took several months to complete the transition in steps, but we’re almost there. It can be done.

I Know What You’re Thinking

You’re thinking that you could never give up the luxury of Visual Studio. You could never survive without the convenience of Intellisense and autocomplete. Your life would be meaningless without your precious breakpoints and line-by-line debugging. You would shrivel up into a dry empty husk in the absence of ReSharper’s loving embrace.

Life with C# on Linux isn’t all that bad. You can still have many of the things you love about Visual Studio, without all of the bloat. The purpose of this article is to explain how I’ve achieved some of that for myself.

Why Make the Switch?

The reasons for phasing Windows and Windows-based tools out of our tech stack can, if you ask me (which you implicitly have by reading this article) be boiled down to “Windows sucks.” The primary motivating factor is that automatically provisioning Windows machines (being able to consistently spin up new instances and get your software and its dependencies installed and configured) is like trying to train a cat to use a human toilet. Sure, it can be done (I’ve seen videos), but it’s difficult, it’s unnatural, and people will think you’re not right in the head for even trying. The fact that Windows servers are typically more expensive (almost comically so when dealing with SQL Server) doesn’t help. Linux is just a cleaner solution for running server infrastructure. Microsoft is certainly headed in the right direction these days, and I have some confidence that they’ll get there eventually, but as it stands today Windows sucks as a server platform.

Windows Server 2012 uses their touch-screen optimized GUI, for goodness’ sake! Who thought that was a good idea? A server operating system doesn’t need a GUI to begin with let alone a poorly conceived tablet-optimized one.

In theory, even if you’re running your production environment on Linux and Mono, you could still choose to develop in Windows and Visual Studio–in my experience, situations in which you need to write code differently between the two platforms are pretty rare and easily avoided. There have been situations we’ve run into where Mono behaves slightly differently (usually in unexpected or broken ways), so if you take this approach it’s possible that you might miss problems with your code that exist in Mono but not Windows, so it’s important to write lots of tests and to run them using Mono prior to deploying.

I am a huge fan of Vim. When I was coding in Visual Studio I was using an extension called ViEmu which added Vim keybindings and features to the editor. The problem with ViEmu is it’s freakin’ expensive. I think I paid $100 up front for a license and then had to re-up my “subscription” a couple times for $50 when new versions of Visual Studio came out in order to get a new compatible version of the extension. I certainly don’t begrudge ViEmu’s creator for making money off his work, but paying hundreds of dollars to use features from a free open-source editor makes baby pandas cry. Add to that the cost of Visual Studio itself plus another “subscription” to ReSharper and you’re starting to reach levels of expense that simply don’t make sense for a solo indie developer (yes, I do this in my day job, but I don’t use the tools they provide for my personal/hobby work on the side). Visual Studio is also slow.

C# Autocomplete in Linux

The simplest option to get intellisense-like autocomplete functionality on Linux is to use MonoDevelop (or Xamarin Studio if you’re on OS X). Meet the new boss, same as the old boss. If you’re happy with a big all-encompassing development environment like this, more power to you! Heck, why not go one step further and use Visual Studio Code?

If you prefer to use your editor of choice (Emacs, Vim, Atom, etc.), you can probably also have autocomplete, it’s just a little more complicated to set up.

Omnisharp and Friends

Omnisharp is a mish-mash of open source stuff that adds .NET tooling to a variety of editors on Linux. For Vim, use your preferred plugin manager to install the Omnisharp-vim plugin. The readme covers the configuration changes and build process required. You will also need to install an autocomplete plugin–there are a few options listed in the readme but the one that I was able to get working is YouCompleteMe. Just make sure to include the --omnisharp-completer argument when running the install script.

I also installed dispatch.vim which will automatically enable the .NET tooling in Vim whenever you open a C# source file. If you didn’t install this, you’d need to manually start up the Omnisharp server and connect to it whenever you want your autocomplete. One downside of the automated method is that you can’t edit files from two separate solutions at the same time. Omnisharp runs as a server that parses your solution and project files, and the autocomplete plugins get their information by querying that server process, which means once that process is started for one solution, opening files from another solution will try to use autocomplete information from the first one. In theory you could work on multiple solutions simultaneously, but it would require manually spinning up and connecting to multiple instances of Omnisharp running on different ports.

Another nice benefit of Omnisharp-vim is that whenever you edit and save a new file, it will traverse up the directory tree and add that file to the first project it finds. That’s handy, but you still end up needing to manually modify csproj files in a lot of situations (adding or updating references, removing or renaming source files, etc.). If there’s something particularly tedious I need to do along these lines, I usually cave in and spin up a copy of MonoDevelop. I would, for example, want to gouge my own eyes out with a rusty spoon if I had to manually update a Nuget package across a couple dozen projects.

Debugging

If you want line-by-line debugging with breakpoints and watches and all that other stuff that you’ve probably become so dependent on in Visual Studio, your only options are MonoDevelop/Xamarin Studio or Visual Studio Code. Personally, I’ve eschewed debugging since making the move to Linux, relying instead on console output and throwing exceptions in my code to help track down tricky issues. At first I thought that this would be a deal breaker, but after several weeks I’ve found that it’s actually not that bad. Being able to set breakpoints and do line-by-line debugging isn’t as important as you think once you get used to life without them.

Testing

Visual Studio has some pretty nice test-running facilities built in. We use Xunit for all of our testing, so I would run all of our tests using the ReSharper Xunit runner. Now I run them all from the command line using the Xunit console runner. Works fine.

One gotcha to be aware of is that if you’re using Xunit 2.0, it doesn’t work in the current version of Mono. There is an outstanding issue in Xunit, but it’s actually due to a bug in Mono. The bug has been fixed, but we won’t be seeing it until Mono 4.2 is released. I have no idea when that will be (as of this writing Mono is at 4.0.3). Fortunately, we were still on Xunit 1.9 when we made the switch, so our tests run fine.

Continuous Integration

I’m going to get really esoteric here and talk about using Shippable to build Mono projects. Don’t do it! It is an exercise in frustration and futility. Something about their setup (which may actually be a larger issue with running Mono in Ubuntu in general) makes Mono unusable. You can read about some of my trials and tribulations in the issue I opened.

Conclusion

This article is a lot more rambly than I had intended, but hopefully I’ve achieved my goal of portraying the current state of working with .NET and C# in Linux as pretty far along–certainly far enough to work competently and comfortably on serious projects. We’re still finding new little bugs and issues in the Mono runtime itself that we have to work around, but no show-stoppers, and things are only going to continue to improve now that Microsoft has open-sourced the .NET runtime.

Welcome to Nashville

I am one of the “fortunate” few who live in one of Comcast’s “trial” markets where they offer a “flexible data usage plan” to their consumers. Okay, I’ll stop with the sarcastic quoting (Sarcastiquotes™) and just call it what it is. Where I live, Comcast essentially has a monopoly and uses that power to fuck over its customers by charging exorbitant overage fees for using the service that they’re already charging exorbitant fees to provide to you.

We have a 300gb monthly bandwidth allowance, and if we go over that they automatically charge $10 per additional 50gb block of data. So if we were to use, as we did a few months ago, 903gb of bandwidth, we would be charged an additional $130 for 13 extra 50gb blocks of bandwidth. Adding that to the $100 we already pay for their 75mbps “triple play” service (a promotional rate we got for the first 12 months of our 24 month contract), plus $10 for the equipment rental fee (we own our own SBG6580 modem and would have been happy to use that, but apparently you’re not allowed to use your own equipment on service plans that include the phone service that we never actually use), plus other various fees and taxes, and we’re paying upwards of $250/month for a service that was advertised at $99/month. Awesome! Not! (Yeah, I grew up in the 90s when saying “not” was invented but still wasn’t funny because it was never actually funny.)

If you live in an area without caps, you may not realize just how infuriating they can be. You may not realize how awful it feels to have to second-guess every little thing you do on the internet. Should I add this file to my Dropbox folder now, or wait until next month when the counter resets? Are my kids still watching YouTube on their iPads? Can I afford to watch another movie off Amazon Prime before the end of my billing cycle? How much is this episode of Game of Thrones going to cost me?

FML. First world problems suck in the first world, right?

AT&T Makes a Funny

A few weeks ago, the doorbell rings and it turns out to be a salesperson from AT&T. Apparently they’re upgrading and expanding their fiber network into our neighborhood, which means Comcast now has some competition in our area. This excites me, but I try to keep my cool. Don’t want the salesperson to get too comfortable. What kind of speeds are they offering, I ask. They have a 18mbps plan and a 45mbps plan, he says. My excitement decreases a bit. How much do the plans cost, I ask. The 18mbps plan starts at $130, he says. My excitement practically evaporates. My problem with Comcast, I tell him, is the 300gb monthly bandwidth cap. I would even be willing, I explain, to pay the ridiculously higher monthly fees that they are asking for the laugh-inducingly reduced speeds that they are offering if it meant I would be free of the burden of monthly caps.

That’s when the salesperson cooly informs me that AT&T also has a 300gb monthly cap on all of their plans, and also charges $10 per 50gb block of additional data.

WHAT A FUCKING COINCIDENCE!

As you can imagine, I politely informed the salesperson that I was not interested in anything more he had to say and could he please get the fuck away from me as quickly as possible before I vomit all over his stupid face. (I actually didn’t say any of that, I’m a pretty tame and polite person in-person. I grew up in Canada, for gosh sake.) Incidentally, I checked online to see what plans/prices they were offering and they were completely different than what the salesperson had told me (but still equally unappealing, including the fact that their cap is listed as 250gb instead of 300gb). I don’t know what to make of that.

The Promise of Google Fiber

Nashville is on Google’s roadmap to get their Fiber service at some nebulous point in the future. I don’t actually live in Nashville. I live in a different city and county about 20 minutes south of Nashville. Far enough that when Google Fiber does come to Nashville, it won’t be an option for me. At least not right away. And even if it was, I don’t know if I trust Google enough to give them full control over my internet connection. My time working in the advertising industry (dealing almost exclusively with Google) has turned me into an extreme privacyphile. I don’t have a Facebook account, I’m not on Twitter, I don’t use Gmail, Chrome scares me, and my default search engine is DuckDuckGo.

I’m pretty much convinced that if Google was my ISP, every byte of data that travelled in or out of my home nework would be stored, analyzed, and possibly altered by Google before reaching its final destination. Why would Google want to store and analyze all of my internet traffic? Fuck if I know. Some evil, money-making, advertising related bullshit, I’m sure. Fuck Google. Fuck Comcast. Fuck AT&T. Fuck me!

Comcast Employs MITM Attacks

To inform their customers when they’re approaching and/or exceeding their monthly bandwidth caps, Comcast helpfully intercepts all incoming HTTP traffic and injects javascript that delivers the message in an inline pop-up alert.

You read that right.

Comcast uses a man-in-the-middle attack to inject pop-up alerts into third party websites that inform their users that they’re now being charged extra for downloading the very website that’s currently being hijacked. Thanks Comcast!

Avoiding the Cap

I’ll cut to the chase here. I switched to a Comcast business plan–50mbps down and 10mbps up for $110 per month. I lose my TV and phone service (neither of which I really used anyway), reduce my download speeds by 33%, and increase my monthly fee by $10. But. No. More. Fucking. Caps. My base monthly fee may be up, but without the caps and overage fees the amount I’m actually paying is over $100 per month less. Add to that the fact that I can use my own modem on the business plan instead of having to rent one means I also save the $10 monthly rental fees.

I never thought I would be grateful to pay (theoretically) more for lower speeds, but Comcast found a way to make that happen.

Taking a Step Back

I know that there are people who would kill to be in my position. I was in Canada for my grandmother’s 90th birthday in May and complaining to one of my cousins (who lives in Ontario) about the state of ISPs and mobile providers where I live. He practically punched me in the nuts. I guess ISPs and mobile providers in Canada are even more bullshit-ier than what we deal with in the US. And I’m sure there are third-world countries in Africanistan or whatever that don’t even have the internet or cell phone service.

For now, I’m content with my cap-free business plan from Comcast, even though I despise Comcast. Until there’s a legitimate alternative with reasonable plans and a respect for privacy, this is probably as good as it’s going to get.

How fucking depressing.

The Setup

A side project I’ve been working on during my free time for a while now is Ficdown, a system for creating choice-based interactive fiction in Markdown. The meat of the project is in the form of a “compiler” that will take a Ficdown source file and generate an epub that can be read on any hyperlink-capable e-reader.

I wrote the compiler in C#, mainly because we coders are nothing if not creatures of habit, and that’s what I’ve spent the majority of my professional career coding in. At work, we’ve recently made the switch from writing and running our .NET stuff on Windows machines and using Visual Studio to doing everything in Mono on Linux machines. Freeing myself from the bloat of Visual Studio in a Windows VM and switching to coding entirely in Vim directly on my Linux host was like a massive breath of fresh air–so much so that I decided to make the same switch at home.

I could (and may at some point) fill a whole post or two about the joys of coding C# on Linux in Vim, but for now suffice it to say it’s everything I love about C#, .NET, and (obviously) Vim, minus everything I hate about Windows and Visual Studio. The best of both worlds. Cake and eating it too. Yadda yadda yadda.

The only thing I was unsure about, as far as Ficdown was concerned, was how I would integrate my compiler into a website without a Windows server to host it on (my current server is a Centos box on DigitalOcean). At work we’re using ServiceStack which has a self-hosting mode, meaning your web app lives inside a console application that does its own connection handling, request threading, and so on. I could have gone that way with the older free version of ServiceStack, or I could try to figure out how to get XSP working with a standard MVC project…

But come on! This is Linux! There are so many cool ways to create websites on Linux that don’t involve maintaining cumbersome csproj and sln files and having to compile and deploy crap. I just want to spin up a light-weight site in something like Node or even a static generator like Jekyll. The only fancy part would be where it needs to post uploaded source files to my C# compiler and then give a link to the generated output.

Enter Edge.js

A static site generator was probably taking it a step too far–I can’t imagine any feasible way to get that going. There’s no client-side .NET runtime written in Javascript that could execute my Ficdown compiler that I’m aware of. Node, however, was a definite possibility. We use Node at work for a bunch of stuff. I’ve worked on some of those projects. I sorta like Node.

Then I remembered about Edge.js.

Now I freakin’ love Node.

Edge.js is basically an interop provider that allows you to run .NET code natively within a Node application (and vice versa). Using that, I could create my website as a simple Node/Express app, and make calls directly to my Ficdown compiler wherever I needed to. So deliciously brilliant! Here’s how I did it:

How I Did It

I’ll skip over the prerequisites since those steps would be ultra boring, change depending on which distro you’re on, and you should be able to work that out for yourself anyway. Basically you need Node, npm, Mono, and (in my case) CoffeeScript installed globally. That’s right, I prefer CoffeeScript over raw JavaScript. Deal with it. Unless you also prefer CoffeeScript, in which case have a high-five!

The web app itself started off about as simply as a Node/Express app can get, which is a thing of beauty.

express = require 'express'
app = express()

app.set 'views', __dirname + '/views'
app.set 'view engine', 'jade'
app.use express.static __dirname + '/public'

app.get '/', (req, res) -> res.render 'index', active: index: true
app.get '/learn', (req, res) -> res.render 'learn', active: learn: true
app.get '/tutorial', (req, res) -> res.render 'tutorial', active: learn: true, tutorial: true
app.get '/reference', (req, res) -> res.render 'reference', active: learn: true, reference: true
app.get '/write', (req, res) -> res.render 'write', active: write: true
app.get '/compile', (req, res) -> res.render 'compile', active: write: true, compile: true
app.get '/playground', (req, res) -> res.render 'playground', active: write: true, playground: true
app.get '/play', (req, res) -> res.render 'play', active: play: true
app.get '/source', (req, res) -> res.render 'source', active: source: true

All of the actions just render their respective views, passing some information that the header uses to highlight menus and stuff. Pretty much a static site. It could probably be done more efficiently but it hasn’t gotten cumbersome enough yet for me to work out how.

The only really interesting view from a back-end standpoint is the compile page, which contains a form that allows someone to upload a Ficdown source file. Here’s essentially what the form looks like (I’m using the Jade templating engine):

form(method='POST' enctype='multipart/form-data' action='/compile')

  p
    label(for='source') Ficdown Source File:
    input#source(type='file' name='source')

  p
    label(for='author') Author Name:
    input#author(type='text' name='author')

  p
    input#debug(type='checkbox' name='debug')
    label(for='debug') Include player state debug information.

  p: input(type='submit' value='Compile')

To wire it up, I created a new file called compiler.coffee and required it at the top of my app with compiler = require './compiler', then added this line to the bottom:

app.post '/compile', compiler

Then I started working on the new handler for posts to the /compile endpoint:

edge = require 'edge'
formidable = require 'formidable'

compile = (req, res) ->
  form = new formidable.IncomingForm()
  form.parse req, (err, fields, files) ->
    if fields.author == '' or files.source.size == 0
      res.render 'compile'

        # user didn't fully fill out the form
        # insert logic to render error on the view here

    else
      data =
        author: fields.author
        debug: if fields.debug == 'on' then true else false
        source: files.source.path

      # need to actually do the compiling here

module.exports = compile

Simple enough so far, especially using Formidable to handle the form parsing for me.

Although, this is where I hit my first snag.

On my machine running Arch Linux, just the mere act of requiring the edge module causes mono to segfault. I’m not sure if it’s because the version of Node or Mono in the Arch repos are greater than what edge has been tested against, but that’s my best guess. I opened an issue about it.

At any rate, to continue I ended up firing up a Centos Docker container with all of the prereqs loaded up in order to continue. The app ran fine in the Centos environment.

Making It Edgier

The Edge docs cover how to integrate .NET code into Node pretty well. Basically for Node to invoke a method in .NET, the .NET function must be of the form Func<object, Task<object>>. In simpler speak, it must be a function that takes an object as its parameter, and returns a Task<object> value (which is basically a function that runs asynchronously and returns an object when it’s done). My first thought was ‘oh great, do I have to add a method with that signature to my Ficdown library now?’ Then I realized how silly that was–Edge supports writing .NET functions and classes inline. I could write the function with that signature right in my compiler.coffee file and have that function invoke the methods already in my Ficdown library as they were intended to be called.

It took a few iterations and different attempts to get my DLL referenced correctly, but here’s what I eventually landed on in my compiler.coffee file:

do_compile = edge.func
  source: ->
    ###
    using System;
    using System.Linq;
    using System.IO;
    using System.Collections.Generic;
    using System.Threading.Tasks;
    using Ficdown.Parser;
    using Ficdown.Parser.Model.Story;
    using Ficdown.Parser.Model.Parser;
    using Ficdown.Parser.Render;

    public class Startup {
      public async Task<object> Invoke(dynamic input) {
        var response = new Dictionary<string, object>();

        // getting the form input from node here:

        var author = (string) input.author;
        var debug = (bool) input.debug;
        var source = (string) input.source;

        // using .NET to read the uploaded file content:

        var storyText = File.ReadAllText(source);

        var parser = new FicdownParser();
        ResolvedStory story;
        try
        {
          story = parser.ParseStory(storyText);
          response.Add("success", true);
          response.Add("warnings", story.Orphans.Select(o => string.Format("Warning (line {0}): {1} {2} is unreachable", o.LineNumber, o.Type, o.Name)).ToArray());
        }
        catch(FicdownException ex)
        {
          story = null;
          response.Add("success", false);
          response.Add("error", ex.ToString());
        }
        finally
        {
          File.Delete(source);
        }

        // generating the epub to a temp file:

        if(story != null)
        {
          var rend = new EpubRenderer(author);
          var output = Guid.NewGuid();
          response.Add("output", output);
          rend.Render(story, string.Format("/tmp/{0}.epub", output), debug);
        }

        return response;
      }
    }
    ###
  references: [ __dirname + '/lib/Ficdown.Parser.dll' ]

compile = (req, res) ->
  form = new formidable.IncomingForm()
  form.parse req, (err, fields, files) ->
    if fields.author == '' or files.source.size == 0
      res.render 'compile'
      # snipped out logic for error rendering
    else
      data =
        author: fields.author
        debug: if fields.debug == 'on' then true else false
        source: files.source.path

      do_compile data, (error, result) ->
        if error?
          res.render 'compile'
          # snipped out logic for error rendering
        else
          res.render 'compile'
          # snipped out logic for success rendering

I removed some of the boring boilerplate code that passes error and success messages to the views for the sake of brevity. Here’s a breakdown of some of the cooler things that are going on here:

  • You write your inline .NET code inside of comments. When the app is run, the code gets compiled. If there are any compile-time errors in your code you’ll see them right away when you try to run your Node app.

  • I add all of the form input values to a Javascript object called data, and then pass that as the parameter to my do_compile function. Edge then marshalls that object into a .NET dynamic object c that I call input.

  • Edge does have a way to marshall binary data between Node and .NET, so in theory I could have read the uploaded file using Node and passed its contents to my function, but I decided it would be easier (and more efficient) to just read the file from the .NET code instead.

  • In the .NET code I construct a response dictionary of values that I want to pass back to Node (was the compilation successful? Were any warnings or errors generated? Where was the epub file written to? etc.) In node that object is marshalled back into a normal JSON object.

Pretty effing sweet.

Conclusion

The final site is a little more complex than what I’ve shared. I still had to add handlers in the Node app for downloading and removing the compiled epub files, and I added in some rate-limiting middleware to prevent abuse of the compile form, but I shared all of the Edge-related stuff here.

Easily spinning up Node-based server apps that can natively hook into powerful compiled .NET libraries is essentially the best thing that has ever happened anywhere ever. I had to pinch myself several times while setting this stuff up to make sure I wasn’t dreaming. I love writing my hard-core libraries in C#, but I hate writing websites in C# (and hate having to host websites on Windows servers). Problems solved.

It’s projects like Edge.js that restore my faith in humanity.

A Dark Past

I’m going to tell you about some stuff I’ve done that I’m not particularly proud of. This happened during a period of my life when I was working for a company in the advertising industry. The company already had a pretty strong handle on the email and display advertising markets, but the team I was hired into was a newer group whose job was to break into the desktop advertising game.

It may not be immediately apparent to you what I mean by “desktop advertising,” but I can guarantee that you’ve run into it at some point before. Every time your Grandma calls you up on the weekend complaining that her computer is running slow, and you fire up her copy of Internet Explorer 7 to find that she’s got twenty different toolbars installed, you’ve encountered the kind of thing my team was working on. Every time you’ve tried to install some open source software through a Google link, didn’t pay attention to checkboxes in the installer, and ended up with half a dozen useless registry scanners, disk cleaners, and so-called “anti-malware” programs unintentionally installed on your computer, you may have me to thank for that.

By way of apology, if you ever meet me in real life, I’ll buy you a beer. Promise. Just please try to resist the urge to punch me. I am very sorry for my involvement in everything that you are about to read.

As morbidly interesting as the desktop side of things might be (I may expand on it in a future post), I’m going to tell you a little bit about what we eventually branched into after the desktop business had settled into a stable channel of revenue for the company. Namely, mobile advertising.

First Attempts

The advertising industry is largely driven by plagiarism–you look for a money-making model that’s working well for someone else, then copy it. If you get in early enough and “drive a truck through it,” as one of my managers used to say, you stand to make a lot of money before rising competition turns it into a race to the bottom and profits dry up. That’s how we approached advertising on mobile at first.

Our first product was an “app-a-day” app for iOS that offered users a free app every day (the implication being that the offered app would otherwise not be free). There was another app called AppGratis that was doing pretty well and we wanted some of that action.

Our app was a flop straight out of the gate. The development philosophy while I was there involved pumping out production-ready products within a day or two–if something was going to take longer than that to get into the wild then it wasn’t worth doing. This meant that most of what we did (including this first iOS effort) was a buggy mess. The idea was that we would throw this low-effort proof-of-concept at the wall and see where it stuck best, then quickly iterate and fine-tune it to maximize profits.

This one didn’t really stick at all. Probably due to the fact that all of our “offers” were games and apps that a) nobody wanted and b) were already free on the app store. We didn’t make any effort to provide actual value to users, and we didn’t provide any value to the publishers because nobody was using our app. The whole thing ended up being moot anyway, because shortly after we got into the App Store, Apple yanked AppGratis and basically banned all “app-a-day” style apps forever. Pay attention and you’ll soon discover that this is the start of a common pattern.

Our struggles with Apple’s iron fist and how long it took to get new changes into the App Store left a sour taste, so we decided to move on to Android. The big money on Android at the time came in the form of push-ads–these were the ads that would appear in your notification center, even when the app that generated them wasn’t running. A company called AirPush more or less had the market cornered on push-ads, so we set out to emulate them and carve out our own little corner.

Since my company already had a vast supply of ads through its email and display channels, it was pretty easy for me to churn out a quick proof-of-concept SDK for Android that would tap our existing ad feeds and push them into the user’s notification center. From there on it became a game of attracting developers to use our network, and optimizing the SDK and ads to maximize profits. It went okay, but developer acquisition was a problem we never really cracked–probably due to our unwillingness to actually put any effort or quality control into anything that we did (improving the quality or “feel” of a product didn’t directly lead to increased profits, so it was generally frowned upon and discouraged).

And then Google dropped the ban-hammer. Push-ads were outlawed. This cut deep enough into profits that it was no longer worth spending time or resources on supporting the ad network, so we basically moved on.

The Collision of Two Worlds

This is when we strayed from the usual path of identifying an existing market to jump into and actually developed something that was, as far as I know, pretty novel. As I mentioned earlier, we had already developed desktop advertising into a thriving channel of the business, so we came up with a way to piggy-back mobile distribution into our existing desktop distribution model.

Once again I pumped out a quick and dirty proof-of-concept–this time in the form of a Windows app–that we would distribute through our desktop installer network as another checkbox for people to miss. This new app would sit in the user’s system tray, silently running in the background.

What did that app do, you ask?

If you have an Android and have spent any time looking for apps in Google’s App Store from your desktop computer, you may have noticed that there is an “Install” button which, when you are signed in, lets you install apps directly on your phone. You click the button on your desktop, the app automagically appears on your phone. You can probably guess where this is going.

Web browsers don’t really do a great job of protecting their cookies on your computer. They’ll go to hell and back protecting them from web-based attacks, cross-site scripting, injected iframes, etc. But once you’re actually on someone’s computer–once they’ve trusted and executed your code–getting their cookies is trivial; IE stores them as a bunch of plain-text files in the user’s directory, and Firefox and Chrome store them in unprotected plain-text SQLite databases (or did at the time, anyway).

So my new little desktop app, which was quickly distributed to millions of unsuspecting checkbox-ignoring users, would “borrow” their existing Google session by reading their browser cookies, then invisibly “click” that App Store install button for them on apps that were paying us for distribution. We started off with opt-in screens and notifications, letting the user know that they have signed up for our free “app discovery” platform and we just sent them a new app, but we quickly learned that if the user became aware of what was going on at any point in the process, they would remove our app and we’d lose them as a user (a-duh!). Over time, those notification and opt-in screens were “optimized” away as much as possible. They already “agreed” to our 23 page EULA when they were trying to install Paint.NET but accidentally clicked the wrong download button anyway, right?

Calling it an “app discovery” platform soon took on a new meaning for us. Usually that’s biz-speak for a service that helps users discover new apps that they want to use, but normally wouldn’t find because they’re buried too deep in the App Store. In our case, it meant users would wake up in the morning and “discover” new apps on their phone with no idea how they got there.

Several of the first apps we pushed were our own tracking apps that would allow us to call home and gather statistics about our users. The nature of the product meant that those apps had to be available through the Google App Store–you can probably imagine what the comments and ratings looked like on those apps. I certainly learned a few new profanities and insults. I also learned how good Google is at banning developer accounts. A particularly low point for me was talking to a Google employee through a newly-generated VOIP phone number under an assumed name, trying to activate a new developer account with a pre-paid credit card and a made-up address several states away. Logging in and managing the developer account had to be done remotely through an Amazon EC2 instance, since our office’s IP address was perma-banned.

It was around that time I started looking for a new job.

No Excuses

The stuff I worked on in that job was complete horseshit. It provided absolutely zero value to anybody. It existed and was expressly optimized for the sole purpose of exploiting non-tech-savvy computer users to generate undeserved profits. We all very much understood that our “users” were generally unaware that they were a source of revenue for us (this was considered a good thing) and it was often joked about. I knew this the whole time I was working there, and I felt shitty about it, but for a couple years not shitty enough to keep me from selling out for a reasonable paycheck, three free lunches every week, and good benefits.

I’ve since moved to a new state, a new job, and a different (less soul-sucking) industry, and feel really, really good about that decision. I’m now working on things that actually provide value to the users. If there’s a moral to this story, I’m not entirely sure what it should be. Maybe that “will it pay the bills?” shouldn’t be your only consideration when exploring new job opportunities. “Could I live with myself?” should be somewhere up there too.

I have no idea if any of the things I helped build are still alive out there. When I left, we still had problems identifying good users to push our desktop app to–it had to be someone who owned an Android, was logged into Google on their desktop, and had enabled the ability to push apps from the App Store to their phone. This is a small segment of the total universe of desktop users, meaning that even though we were able to make insane amounts of money off the users we got, we weren’t able to get that many users. With the never-ending Google account closures to boot, it wouldn’t surprise me if that product was eventually tossed into the heap along with our other failed endeavors to make way for the next million-dollar-idea. Though, the last thing they had me working on before I left was reverse-engineering how iTunes installed apps in the hopes of developing a similar distribution model for iOS. We knew it was possible because there were a couple Chinese products out there that could push signed apps directly to iOS devices already.

I’ll end this post by once again apologizing for everything I did while working there. It is a definite fact that I made thousands (at least) of peoples lives a little bit worse through my efforts, and that still bugs me. But that’s okay–hopefully it means I managed to escape with my conscience still somewhat intact.

So please, tell your Grandma I’m sorry. And to upgrade her browser.

The Setup

At work we’re currently in the process of migrating our larger API project from a Windows host running under IIS to a Linux host running under Mono. The project is written using ServiceStack, which means we can use its built-in self-hosting mechanism instead of having to run under XSP, which is really nice. We already successfully made the transition with a smaller API project several weeks ago, so we were able to apply the things we learned during that migration to avoid many of the trials and tribulations involved with getting an existing .NET project to compile and run under Mono this time around. I may write more about that process some time in the future, but that’s not what this post is about.

This post is about something much more specific and unlikely to be of interest to almost anyone except me, and maybe some hypothetical programmer who finds themselves in the exact same scenario and happens to stumble across this post after tearing his or her hair out all night. To that hypothetical programmer, you are welcome. To everyone else, I am sorry. Unless you actually do find this post interesting for some reason, in which case you are also welcome.

Working in the healthcare industry means having to be HIPAA compliant, which is basically a set of rules and guidelines intended to keep personal health information protected from prying eyes (that’s probably grossly oversimplifying it, but you get the idea). Part of our service involves storing sensitive information in S3 which, in order to be HIPAA compliant, has to be encrypted. We also have to decrypt when an authorized user wishes to retrieve it. Fortunately, Amazon has an official AWSSDK library in Nuget which makes communication with S3 a breeze, and .NET has a ton of standard crypto functionality built in which makes encrypting and decrypting a breeze. Here’s the cool part:

The Cool Part

Amazon’s S3 SDK deals with streams when you upload and download, and .NET has this thing called a CryptoStream which lets you encrypt and decrypt streams on the fly. Nifty! An astute programmer might put two and two together and realize that he or she could handle uploading by passing the stream directly from the client into a CryptoStream and pass that to the S3 SDK in order to encrypt the data as the user uploads it and pass the encrypted bytes directly for storage in S3. That way the whole process is completed without ever storing the sensitive information anywhere on the server’s disk, or even holding it all at once in memory. Similarly, downloading could be handled by passing the encrypted bytes from S3 to a CryptoStream and then passing that directly on to the client, again never having to store any information on the server’s disk or in memory.

Here’s some example code to show just how slick the upload could look:

public void EncryptAndUpload(Stream stream, AmazonS3Client s3Client,
  string bucket, string key)
{
  using(var aes = new AesManaged
  {
    BlockSize = 128,
    KeySize = 256,
    Mode = CipherMode.CBC
  })
  {
    aes.GenerateKey();
    aes.GenerateIV();

    using(var encryptor = aes.CreateEncryptor())
    {
      using(var cstream = new CryptoStream(stream, encryptor, CryptoStreamMode.Read))
      {
        s3Client.PutObject(new PutObjectRequest
        {
          BucketName = bucket,
          Key = key,
          InputStream = cstream
        });
      }
    }
  }
}

Sweet, right? You pass it your stream and the S3 client and it encrypts the stream and stores it in S3 for you. Instead of reading all the unencrypted bytes into memory, encrypting them in memory, then sending all the encrypted bytes to S3 like some noob, we’re getting all streamed up in that shit. Stream-style baby.

Wrong.

If you try running this, you’ll get an exception because the AmazonS3Client wants to know the length of the stream before you upload it, and CryptoStream.Length throws an exception (it doesn’t know how to guess the length of the encrypted stream before it’s been encrypted). Another problem that may arise, even if this did work, is that you have no way of verifying that the encrypted content in S3 hasn’t been tampered with when you come back to decrypt it later. Well, wipe away those tears my friend, we’re not finished yet.

Two Birds with One Stone

CryptoStream may not know how to calculate the final size of its encrypted bytes, but we do! We are specifying an AesManaged.BlockSize value of 128 bits. This means the final stream will be encoded into chunks of 128 bits (or 16 bytes). The final size of the stream will be the minimum number of 16-byte chunks that would be required to hold the original stream. You can calculate that from the original stream using the formula streamLength + blockSize - (streamLength % blockSize). Trust me, I got that off StackOverflow so it must be right.

Another nifty feature of Amazon’s S3 is that it stores the MD5 hash of the data stored at a particular key as metadata–namely the ETag value. If we could calculate the MD5 hash of our encrypted data locally, we could use that value to check against the ETag in S3 to make sure they match later before we decrypt it.

So, if you’ve been paying attention, you’ve probably figured out that what we want is a way to wrap our CryptoStream with a facade that can a) report the length of the final encrypted stream and b) calculate the MD5 of the encrypted data as it’s being read by the AmazonS3Client.

public class MD5CalculatorStream : Stream
{
  private readonly Stream _stream; // the stream to wrap
  private long? _length; // stream.Length override
  private bool _firstPass; // we only need to calculate the MD5 once
  private readonly MD5CryptoServiceProvider _md5 = new MD5CryptoServiceProvider();

  // constructor takes a stream and an optional pre-calculated length override
  public MD5CalculatorStream(Stream stream, long? length = null)
  {
    _stream = stream;
    _length = length;
    _firstPass = true;
    _md5.Initialize();
  }

  // property to access the MD5 after it's been calculated
  public byte[] MD5
  {
    get { return _md5.Hash; }
  }

  // pass SetLength calls to our override if we have one
  public override void SetLength(long value)
  {
    if(_length.HasValue)
      _length = value;
    else
      _stream.SetLength(value);
  }

  // here's the meat and potatoes, calculate the MD5 as the stream is read
  public override int Read(byte[] buffer, int offset, int count)
  {
    // calculate the MD5 in blocks as the stream is read
    var bytesRead = _stream.Read(buffer, offset, count);
    if(_firstPass)
    {
      _md5.TransformBlock(buffer, 0, bytesRead, null, 0);

      // if that was the last block, finalize the MD5 hash
      if(bytesRead < count)
      {
        _md5.TransformFinalBlock(new byte[0], 0, 0);
        _firstPass = false;
      }
    }
    return bytesRead;
  }

  // return our length override if it exists
  public override long Length
  {
    get { return _length ?? _stream.Length; }
  }

  // amazon also calls this for some reason so we need it to not throw errors
  // doesn't seem to matter that the values we return are incorrect
  public override long Position
  {
    get { return _length.HasValue ? 0 : _stream.Position; }
    set { if (!_length.HasValue) _stream.Position = value; }
  }

  // override the rest of Stream's members, passing the calls directly to _stream
  // ...
}

The above class does what we want–it can report a stream length that you give it beforehand, and it uses the MD5CryptoServiceProvider to calculate the MD5 of the stream as it’s being read. I didn’t complete the implementation, which involved creating simple wrapper functions for the other members of Stream such as Stream.Flush, Stream.Write, etc. For these you simply call those same functions on _stream in order to pass them through. The exceptions may be the Seek related methods, since seeking will screw up the MD5 calculations, you probably don’t want to let anyone do it on one of these streams. Up to you. AmazonS3Client doesn’t do any seeking so it doesn’t matter either way for the purposes of this article.

Here’s what our new upload function looks like:

public void EncryptAndUpload(Stream stream, AmazonS3Client s3Client,
  string bucket, string key)
{
  using(var aes = new AesManaged
  {
    BlockSize = 128,
    KeySize = 256,
    Mode = CipherMode.CBC
  })
  {
    aes.GenerateKey();
    aes.GenerateIV();

    using(var encryptor = aes.CreateEncryptor())
    {
      // calculate the future length of the encrypted stream
      var clen = stream.Length + (aes.BlockSize >> 3)
        - (streamLength % (aes.BlockSize >> 3));

      using(var cstream = new MD5CalculatorStream(
        new CryptoStream(stream, encryptor, CryptoStreamMode.Read),
        clen))
      {
        s3Client.PutObject(new PutObjectRequest
        {
          BucketName = bucket,
          Key = key,
          InputStream = cstream
        });
      }
    }
  }
}

Voila. Shit just works now. Here’s what the download function looks like:

public Stream DownloadAndDecrypt(AmazonS3Client s3Client,
  string bucket, string s3key,
  byte[] key, byte[] iv, byte[] md5)
{
  var get = s3Client.GetObject(new GetObjectRequest
  {
    BucketName = bucket,
    Key = s3key
  });

  // make sure the ETag matches our MD5
  // if not, the data in S3 changed since we encrypted it!
  var md5str = BitConverter.ToString(md5).Replace("-", string.Empty);
  if(!get.ETag.Trim('"').Equals(md5str, StringComparison.OrdinalIgnoreCase))
    throw new Exception("OH CRAP WE'VE BEEN HACKED ALERT THE CEO!!");

  using(var aes = new AesManaged
  {
    BlockSize = 128,
    KeySize = 256,
    Mode = CipherMode.CBC,
    Key = key,
    IV = iv
  })
  {
    var decryptor = aes.CreateDecryptor();
    return new CryptoStream(get.ResponseStream, decryptor, CryptoStreamMode.Read);
  }
}

Bam! This function first verifies that our MD5 matches the ETag from S3, then sets up a CryptoStream to decrypt the data which can then be passed directly to the client. No problemo! Done, right?

Wrong.

If you’re on Windows, then yeah, you’re done. In Mono, as of the date of this post, you are not done. There’s some bug somewhere in either the AWSSDK or in the Mono runtime that causes the CryptoStream to throw an exception on the final block as it’s decrypting.

Why I Cried Myself to Sleep that Night

I spent a long time trying to figure out a clean way to work around this problem, but in the end I had to move on in to Noobsville.

public Stream DownloadAndDecrypt(AmazonS3Client s3Client,
  string bucket, string s3key,
  byte[] key, byte[] iv, byte[] md5)
{
  var get = s3Client.GetObject(new GetObjectRequest
  {
    BucketName = bucket,
    Key = s3key
  });

  // make sure the ETag matches our MD5
  // if not, the data in S3 changed since we encrypted it!
  var md5str = BitConverter.ToString(md5).Replace("-", string.Empty);
  if(!get.ETag.Trim('"').Equals(md5str, StringComparison.OrdinalIgnoreCase))
    throw new Exception("OH CRAP WE'VE BEEN HACKED ALERT THE CEO!!");

  using(var aes = new AesManaged
  {
    BlockSize = 128,
    KeySize = 256,
    Mode = CipherMode.CBC,
    Key = key,
    IV = iv
  })
  {
    // lame crappy work around hack that makes me sad
    var dstream = new MemoryStream();
    get.ResponseStream.CopyTo(dstream);
    dstream.Position = 0;

    var decryptor = aes.CreateDecryptor();
    return new CryptoStream(dstream, decryptor, CryptoStreamMode.Read);
  }
}

As you can see, to get it working, all that’s required is to read all of the encrypted bytes from S3 into memory first and then pass that to the CryptoStream. But my vision of an all-streaming solution has been shattered. While I’m happy that this is finally working, the inelegance of having to store the whole thing in memory has left me a broken man. I have failed.

Moral of the Story

Never give up. Never surrender. Unless there’s a bug in a third party SDK or the framework, then you’re shit-outta-luck. I suppose you could track down the actual bug, fix it, and try to open a pull request into the official repo, but ain’t nobody got time for that.

Hi and welcome to The Code Word. I’ll be using this blog to document various things I find interesting related to coding and the tech industry in general. I’ve been an active software engineer for over a decade now, currently working primarily in C# using Mono, with a little Javascript sprinkled in to keep things annoying. Lots of my peers use something called Python, which I think is like Perl, but not as powerful, or good.

Most posts here will have comments enabled which will allow you to leave feedback and correct any errors I made.

I should note that this is a personal blog, and the opinions stated herein are mine and mine alone. They do not reflect the opinions of my employer (assuming I’m currently employed). If you have an issue with something on this blog, you can take it up with me directly. My email should be in the sidebar.