DSLs

DSL Parser - Sample Code

I while back I wrote a blog series about DSLs, grammars, tokenizers, parsers and a SQL generator. The idea was that you could write a DSL query to mine your error log data and the code would generate SQL. The series can be found here: http://jack-vanlightly.com/blog/2016/2/3/how-to-create-a-query-language-dsl. 

The tokenizer while simple was very inefficient, so I wrote a better one, you can find that here: http://jack-vanlightly.com/blog/2016/2/24/a-more-efficient-regex-tokenizer

I have just published working code based on this series and the better Regex tokenizer on Github here: https://github.com/Vanlightly/DslParser.

A More Efficient Regex Tokenizer

As part of a DSL parsing series, I wrote a post about a super simple yet memory inefficient way of tokenizing some input text. The benefit was that the tokenizer was extremely simple but the downside was that it wouldn't be suitable for large texts or if the tokenizer was called excessively.

In this post we'll look at a similar Regex based tokenizer and trade-off a little simplicity for lot of performance gain.

Implementing a DSL Parser

We previously created a tokenizer that breaks up a sequence of characters into a sequence of tokens (enum TokenType) and a LL2 production notation grammar that acts as a template for the code of the parser.

The input of this parser will be the sequence of tokens and the output will be an Intermediate Representation (IR) which is a data structure that represents the DSL text in a structured manner. The next step, after parsing, will be translating this IR into SQL.

Creating a Simple Tokenizer (Lexer) in C#

What is a Tokenizer?

Before you read on, there is a more efficient version of this tokenizer here: http://jack-vanlightly.com/blog/2016/2/24/a-more-efficient-regex-tokenizer that has significantly better performance.

So a tokenizer or lexer takes a sequence of characters and output a sequence of tokens. Let's dive straight into an example to illustrate this.

Meet a simplified version of Logging Query Language (LQL)

How to Create a Query Language DSL with C#

Creating your own DSL is fun, it involves multiple complex steps which can be challenging and very rewarding to figure out. However if you don't know what you're doing then your code can end up one big hack which, while works, is complicated, hard to change and hard to read. Although pleased that you have a working DSL you know that underneath it's no looker. In this series I will go through, step by step, creating a simple query language that gets translated into SQL and executed against SQL Server.

Logging Query Language (LQL) – a DSL for Log Analytics

LQL came about because of a log analytics platform that I developed for Vueling Airlines. Being a backend developer and a self-confessed newbie at front end (HTML, CSS and Javascript), I originally created LQL in order to avoid having to develop complex front end code for generating queries. I decided that instead I would create a Domain Specific Language (DSL) that would allow users to write the queries and all I would need to provide was a text area.