Going Academic

Dear (3) readers... I have done it, I've written a crappy paper! Now don't get me wrong, I believe the work is worthy but it is my first and the writing may crap the bed, so to say.  Here's is where you come in and peer review the living feces out of it, hopefully.

Here is the link: https://github.com/xs-admin/Excess/blob/master/Doc/Excess.pdf

Compliments are great and I'll take them, but the real useful stuff is if you get real mean and tell me why none of that works.

The case for business

Let's now make the case for how Excess can help businesses. Software companies, particularly the large ones, seldom need GREAT code. What they mostly need is consistent code: their main problem turnover. One senior developer leaves and whole products get shaken to their foundation. If no one understand that dude's code the poor old manager can see months of refactoring coming a mile away. And these things cost many-many-monies.

So, the large company yearns for uniformity. They build thick handbooks (like, 400 pages long) of coding standards, best practices and all kind of minutiae. Problem is, not only those are difficult to learn, they are difficult to enforce. So here comes countless hours of either code reviews or praying. And they pay by the hour.

It is safe to say companies would be delighted if a computer wrote most of their code. Coincidentally that's what excess do, write less code and let the compiler take care of boiler plate. Lets see a user case of a product that has gained some notoriety: aspboilerplate.net

The premise is simple: you will get code generated for you already embedding the best practices. It is a great idea, and I personally used it to create metaprogramming.ninja, It has some drawbacks, tho: It only helps while creating your project, you are expected to follow guidelines from there on. So lets take that to the next level. For the sake of argument lets assume you want to standardize your data access with ORM entities and repositories supporting unit of work and all that jazz:

entity Product
{
     int ID;
     string Name;
     decimal Price;
}

repository Products
{
      IEnumerable<Product> GetAllProducts();
}

By now you have got to know Excess can EASILY generate the boilerplate to make this happen, including units of work and whatever else you need. So we'll talk about subtler points: Let's assume your best practices include not littering your data entities with methods. Excess can report errors when that happens! No need to code-review for that particular item!! In other words, Excess will help your company to add its own best practices to the compiler. Now programmers can come and go, and data access will be standardized, or it wouldn't compile.

Lets analyze another terrible occurrence for business: technology has evolved! Do you know how expensive a system-wide porting can be? Depending on the size of the code base and how old the former technology was it could take from 6 months to many years. Then you are paying your devs for basically nothing.

Lets say that if Excess can generate boiler plate for one technology it could easily do it for another, no? So, want to switch from EntityFramework to Hybernate? Change the underlying data compiler and you are good to go, no changes in the actual application code. Monies people, money!

One last point, we have discussed here a common data extension, however, for some business operating in a well defined domain the savings could be much greater. A criticism could be made as developers must learn a different dialect of the language. But, would that be any different than learning heavy "best practice" handbooks?

Getting real useful (Dudeism edition)

Here's the scenario: you are a programmer, you know, like everybody else. You are writing your project or your task at work and you are generally happy about the state of things. And then you need some statistical work. Now, like most dudes/dudettes, you forgot the courses you took at school. Because, I mean, cmon...

HOWEVER, what you lack in respect for educational institutions you make up for with your general knowledge of the tools available. You know there is a programming language named R for exactly that type of work, with a bunch of ready to use examples and the such. Unfortunately, being the knowledgeable individual you are you know that you either write your whole thing in R or none of it.

So, you put your head down and head for stackoverflow to find a list of available libraries, most of which will be poorly documented and start the long, tedious process of learning the thing. Like you were some kind of nerd, you know? Like you were going to go for a phd in statistics or something.

And then, all of a sudden, things don't look so peachy anymore and you are one bummed out dude/dudette. All you wanted to do was:

private void calculateTheStuff(data)
{
    var result = R()
    {
         //do the amazing in R...
    }

    return result;
}

My calling in life is, as widely known, to help the dudes of the world being generally happy about the state of things. So, basically, this sample code is now possible. And by "basically" I mean, no, you cannot write full R yet. But there IS a nascent implementation of it here, It does the basic vector configuration and concatenation. It is the prettiest thing ever but needs some dudes to help out.

Now, if you are a manager reading this and wondering why would you care about those dudes. Think that studies have clearly linked the bummed-out index of your dudes to major loss in productivity, missed deadlines and the such. You know, the kind of stuff that gets you fired, so, for crying out loud...

HELP YOUR DUDES.  

We've got grammars

It's been a couple of weeks, release time! Our new feature is custom grammars, or the ability for users to embed their languages (or existing ones) in their favorite language. The implications of this feature are clear: domain specific languages can now be a seamlessly integrated into mainstream languages.

Much  have been written about DSLs, with the consensus been:

a) These languages are incredibly useful.
b) These languages are incredibly hard to be made useful.

Just think for a second the amount of work it would take to write one such language and make it mainstream. Now compare that mental image to this:

Now, I know this programming language stuff is hard, so I apologize. It should actually take more than 2 minutes to create an useful extension like this one. However, clocking in at 125 lines of code, I assume an afternoon is enough, Not bad for something you wouldn't have otherwise.

What about the grammars, tho? Well, the grammars used are strictly ANTLR grammars. For the uninitiated, antlr would be the de facto standard for grammar writing. In fact, we do not even process the grammar themselves, letting their compiler generate the basic c# infrastructure. This makes the whole thing very tested and solid. Thanks, @antlrguy.

What we do, after the grammar is processed, is to augment the generated code so it calls your transform functions. As seen in the video, we can generate these functions automatically for you. All you have to do is provide the right transformations.

These transformations are in essence the same as traditional Roslyn transformations. Only the starting point is not a SyntaxNode, but a node derived from the grammar in question. In  this particular example the pairs [name, value] are transformed into a c# anonymous object. Straightforward stuff.

A ninja is born

Look ma, it is a ninja! and he/she does metaprogrammin'!!!

But ma, why so ugly? Well honey, he's made of programmer art, it is very meta. He also talks about fruit and stuff:

Programming languages are measured largely by their expressiveness. If the problem you are trying to solve deals with apples, a programming language made of oranges will add unnecessary complexity. Historically, language makers deal with this problem by adding as many fruits to their language as humanly possible. Which still fails when you a get a problem related to vegetables. Vegetable people are forced then to put together a bunch of fruits and try to make it look like veggies. These are called libraries, which are forced to sacrifice a lot of expressiveness because of the strong smell of fruit imposed by the owners of the language. Some others have it worse, because their problems are related to spaceships and there is no way of making one of those out of fruit, unless millions and millions of fruit are used.

And so, the new compiler is already up, and the brand new address is:

 www.metaprogramming.ninja

There is also documentation at: https://github.com/xs-admin/Excess/wiki with a bunch of examples and stuff.

So, somebody use it now!!

The shape of things to come

Our new compiler is here!! The stats:

• Compiler: 3500 loc
• Extensions: 1000 loc, 15 extensions, about 66 loc per 
• Tests: 12 passed
• Dev time: 3 weeks flat
• Awesomeness level: not for me to say (the awesomenest level)

But first thing first: what was wrong with the old compiler? A lot, what was there was basically me learning Roslyn and amalgamating features whether they were round or square. In other words, the beauty of writing prototypes: they show you all that was wrong with your initial thinking.

And the main thing I learned was not to be overly reliant on Roslyn. Excess is at its core a substitution engine: write the extension, wait for the compiler to complain, modify the syntax tree of the extended code so its valid C# and Roslyn is all smiles.

Problem is, Roslyn is inconsistent detecting defects. And don't get me wrong: Roslyn tries real hard and its absolutely awesome at this. But it still goes haywire in certain cases and in ways that make the extension writer's job a living hell. It is all about patterns: If you can get Roslyn to report your extension's errors consistently you are golden. But sometimes it is awful, take for example the current match extension up in the website:

           match(x)
           {
               case 0: do_something();
               case > 10: do_something_else();
           }

In this case Roslyn scoffs at cases without a matching switch and produces a stream of nodes that makes little sense and forces a lot of parsing chasing expressions and whatnot. We were promised better. The solution, obviously, is to help Roslyn a bit by feeding it a parse tree with less distortion.

What if we could simply substitute the match keyword with a switch and then process it syntactically? That would certainly make Roslyn all warm and fuzzy and we could be processing a sound, predictable parse tree and hence we would live happy ever after. Well, fear no evil my friend:

            var lexical = compiler.Lexical();

            lexical
                .match()
                    .token("match", named: "keyword")
                    .enclosed('(', ')')
                    .token('{')
                    .then(lexical.transform()
                        .replace("keyword", "switch")
                        .then(ProcessMatch));

This shouldn't require much explanation: during the lexical pass (i.e, before compiling a parse tree), match a pattern corresponding to our match keyword (no relation). When found, replace the keyword with "switch" and only then call ProcessMatch. Which is s Roslyn function taking a switch statement and transforming it into a series of ifs. You can  see that code here, much simpler and solid.
 
The first thing to notice is the process is now completely declarative, you tell the compiler what to do without having to worry too much about how is done. Very easy to read and understand. The plumbing (i.e, tree transforming) get confined to specific methods and only deal with specific parts of the tree.

Second: the compiling process is now divided into passes, first you transform the token stream (lexical), then the syntax tree and finally you apply semantically-derived changes where needed. And the syntax you use is consistent throughout your entire extension. For instance, a constructor construct looks like this:

            sintaxis
                .match<MethodDeclarationSyntax>(
                    method => method.ReturnType.IsMissing &&                                                                                        method.Identifier.ToString() == "constructor")
                .then(ProcessConstructor);

Or: find me a method missing its type and with the name "constructor" and apply it a transformation. It don't get any simpler, does it? The result of all this is my old xs language being written in about 600 loc, including the following extensions to C#:

1- JS-style function as lambdas 
2- JS-style function as method, with return type inference 
3- JS-style function inside of code blocks 
4- function as types 
5- method construct with return type inference and automatic visibility 
6- property construct, with type inference, automatic visibility, initialization 
7- constructor construct 
8- Delegate-less event declaration 
9- Syntactic event handlers: on click(args) {} 
10- JS-style arrays: var x = [1, 2, 3]; 
11- typedef construct, supporting generic parameters, etc. 2 variations. 

But more importantly, that the ability of users (not owners) to extend their languages of choice is real, and real easy. 

match, rc3

and that was that, last commit/release for the night, needs more testing and loving. But then again, who doesn't. The match in the name stands for a fairly sweet new dsl I put together, the idea being having a sample extending code.

http://xslang.azurewebsites.net/#/project/4

The match construct works as an extended switch the very basic usage so far implemented looks like this:

As you can see, expressions are permitted in the case statement. Including, the > 10 in line 11. Which is pretty sweet, one must admit. In its final form, this dsl will be so much cooler. For instance, match could return a dynamic value. Or use continue and break creatively.

Fun times.

As for the rc3, it means there is a semi-usable version somewhere in the cloud. All samples running and its behaving decently. All in all not a bad day for the unemployed.

So goodnight.