R0.1.2 | Injecting Front Matter Syntax Highlighting in Markdown Files

TL;DR Contributing to Visual Studio Code Timeline

In continuation of, Microsoft Visual Studio Code: First Contribution, I’ll be talking about my experience working with Visual Code extension grammars, Text Mate, and extension development.

Recap

Goal: implement a feature in a Visual Studio Code that will highlight a TOML Front Matter code block within a Markdown file.

Challenge: TOML isn’t natively supported in VS Code and requires extension support. Compared to an integrated language (ex.YAML) can be referenced by its source file source.yaml within markdown.tmLanguage.json.

First I’d like to simplify a process that took me quite some time to understand.
Exactly how syntax highlighting rules are injected into the editor.

1. A VS Code extension creates a language server in a separate process.
2. The extension host is actively listening for specific file types.
3. A Text Mate, `tmLanguage` file defines valid syntax (patterns) specific to the language. For example, the JSON snippet below will detect the following code block:

   //see below for definition
   ```superjs
   ```
   

   "superjs-code-block": {
   			"begin": "superjs",
   			"end": "(^|\\G)(?=\\s*[`~]{3,}\\s*$)",
   			"contentName": "meta.embedded.block.superjs",
   			"patterns": [
   				{
   					"include": "source.js"
   				}
   			]
   }
   

4. When a pattern is detected, VS Code uses the appropriate colours to highlight the syntax depending on the current theme.

First Attempt

Although I had opted to go the more practical approach, I felt understanding how the “simpler” method is implemented could get me off to the right start.

Easy Fix: Visual Code has native support for various languages such as Markdown. The markdown.tmLanguage.json contains tons of rules for syntax highlighting within a Markdown file. Create a new rule that would be triggered as long as the user had TOML lanugage support installed.

Looking through the markdown.tmLanguage.json file you’ll notice many fenced_codeblock rules.

{
    "include": "#fenced_code_block_xml"
},
{
    "include": "#fenced_code_block_xsl"
},
{
    "include": "#fenced_code_block_yaml"
},

This naming convention was used for standard Markdown fenced code blocks using triple backticks ```. Although this wasn’t quite what I needed, it gave me a better understanding of how the XML elements were structured.

I continued to search through the file and came across another XML element named frontMatter.

"frontMatter": {
    "begin": "\\A-{3}\\s*$",
    "contentName": "meta.embedded.block.frontmatter",
    "patterns": [
        {
            "include": "source.yaml"
        }
    ],
    "while": "^(?!(-{3}|\\.{3})\\s*$)"
},

The above pattern defines a valid Front Matter YAML code block which uses triple hyphens ---.

My plan was to duplicate the YAML Front Matter rule and modify the regular expression to check for triple addition symbols +++. While trying to import source.toml.

I compiled and launched VS Code source with my changes only to find out nothing really changed.

Issue#1 – Since source.toml is not integrated withing VS Code there is no way of refrencing it’s source.

Second Attempt

At this point, I was getting frustrated with my lack of progress. Endless Google searches and document after document I decided to start over with a new plan. This entire time I was focused on the bigger picture. Instead of trying to integrate the feature directly into the VS Code dev environment, why not create a simple extension with the sole purpose of injecting a code block.

Duplicating Matt Bierner’s original demo, I would be able to test each function as I went along.

I added following lines to support the new code block.

Line 3: “injectionSelector”- The L represents the left side of a line of text within a markdown file.

Line 6: “include” – the name used to define your new rule on Line 10.

Line 11: “begin” – regular expression looks for the three consecutive addition symbols +++ on the first line of the file with no leading whitespace.

Line 12: “end” – regular expression looks for the three consecutive addition symbols +++ to close the block.

{
	"fileTypes": [],
	"injectionSelector": "L:text.html.markdown",
	"patterns": [
		{
			"include": "#toml-code-block"
		}
	],
	"repository": {
		"toml-code-block": {
			"begin": "\\A\\+{3}\\s*$",
			"end": "(^|\\G)(?=\\s*[\\+~]{3,}\\s*$)",
			"contentName": "meta.embedded.block.toml",
			"patterns": [
				{
					"include": "source.toml"
				}
			]
		}
	},
	"scopeName": "markdown.toml.codeblock"
}

I tried creating a copy of TOML.tmLanguage within the projects directory, in hopes that I would be able to reference it by scope.

Line 5: “path” – path to the TOML.tmLanguage copy

Line 11: “injectTo”: “text.html.markdown” – explicitly declares where this rule will be used.

 "grammars": [
      {
        "language": "toml",
        "scopeName": "source.toml",
        "path": "./syntaxes/TOML.tmLanguage"
      },
      {
        "scopeName": "markdown.toml.codeblock",
        "path": "./syntaxes/codeblock.json",
        "injectTo": [
          "text.html.markdown"
        ]
      }
    ]

Testing the newly created extension, it was still unable to load TOML.tmLanguage file. BUT I was able to reference other syntax source files (source.js) to demonstrate that the new pattern was working.  Notice how the labels below such as name have no syntax highlighting.

Issue#2 – Although the tmLanguage file path is set (line 5: package.json) , VS Code still requires a grammar server to access source.toml

After reading over VS code’s “Extension Authoring” documentation, they had referenced a tool named Yo Code.
A Yeoman tool that generates basic Visual Studio extension templates for themes, language support, snippets, TypeScript and JavaScipt extensions.
During this setup process, I was able to directly reference the toml.tmLanguage file from the “TOML Language Support” extension.

I couldn’t believe how simple the setup process was. It generated the necessary files to access a TOML language server.

Success

I rebuilt the extension along with the newly integrated language server, and to my surprise…

A valid Front Matter block on the first line of a Markdown (.md) file with no leading whitespace. Also, the code block can’t interfere with other syntax rules.

Contributing

Midway through the process, I realized that I hadn’t submitted a single pull request.

Issue#3 – How do I contribute this thing?

Typically you’ll want to submit a somewhat working version of your code for review. Considering this feature would be injected from an extension, I was confused as to what the protocol was.

So I reached out for some feedback:

The better-toml extension was created and maintained by a single person, bungcip. With minimal activity in recent months, I was concerned about if and when I would receive a response.

At this point, I did what I could and opened an issue. I made sure to clearly outline all the details and reference the original issue from VS Code.

Within a day I had received a response from bungcip. Since I had uploaded a demo repository, he was able to download it, test it and provide feedback. He had suggested some minor changes and outlined his requirements for structuring the pull request.

Within a few days, the changes were approved and better-toml 0.3.2 was released to the Visual Studio Marketplace and it’s 36,000 users.