Do You Want to Build a

Build Tool?

Do You Want to Build a

Build Tool?

Hi Everyone, I'm Craig !

Hi Everyone, I'm Craig !

  • twitter.com/phenomnominal
  • github.com/phenomnomnominal
  • linkedin.com/in/craig-spence

@phenomnominal 2021

@phenomnominal 2021

Today, I'm going to tell you a fairy tale!

Today, I'm going to tell you a Fairy Tale!

@phenomnominal 2021

Ready?

Ready?

@phenomnominal 2021

It was Summer in the Kingdom of Arendelle!

It was Summer in the Kingdom of Arendelle!

@phenomnominal 2021

And Princess Anna was So Excited!

And Princess Anna was So Excited!

@phenomnominal 2021

The day had finally arrived!

The day had finally arrived!

@phenomnominal 2021

Queen Elsa, Anna's sister and Best friend

Queen Elsa, Anna's sister and Best friend

@phenomnominal 2021

Had finally agreed to share her secrets!

Had finally agreed to share her secrets!

@phenomnominal 2021

Anna, and her friends Kristoff, Sven and Olaf

Anna, and her friends Kristoff, Sven and Olaf

@phenomnominal 2021

Had been working hard all winter

Had been working hard all winter

@phenomnominal 2021

And now it was time for their reward!

And now it was time for their reward!

@phenomnominal 2021

Elsa has Amazing powers!

Elsa has Amazing powers!

@phenomnominal 2021

She can sense the water molecules around us,

She can sense the water molecules around us,

@phenomnominal 2021

transform those water molecules into ice,

transform those water molecules into ice,

@phenomnominal 2021

And control that ice however she likes!

And control that ice however she likes!

@phenomnominal 2021

today, Elsa IS going to share her magic!

today, Elsa is going to share her magic!

@phenomnominal 2021

Chapter One

Chapter One

Just like magic

Just like magic

"Anna, do you remember when we went on our trip into the internet?",

asked Elsa.

@phenomnominal 2021

"Of course!",

"We met all the other princesses, it was so much fun!"

replied Anna.

@phenomnominal 2021

"There was that cute, new  little princess, Vanellope!"

@phenomnominal 2021

"And her friend Ralph was in trouble!"

@phenomnominal 2021

"And we helped save him!"

@phenomnominal 2021

"That's right! It was such a fun trip!"

"What else do you remember about the internet?"

@phenomnominal 2021

"There was so much to do, so much to see!",

"So much data flying around! l learned all about how to make websites!"

exclaimed Anna.

" l loved it so much l even taught Sven when l got back!"

@phenomnominal 2021

*SLURP! *

, Sven confirmed.

@phenomnominal 2021

"l learned to write HTML,

and CSS,

and JavaScript!

"lt was just like magic!"

@phenomnominal 2021

It was

just

like

magic

@phenomnominal 2021

Chapter Two

Chapter Two

Sufficiently incredible

Sufficiently incredible

Anna, Kristoff, Olaf, and Sven waited patiently for Elsa to continue.

@phenomnominal 2021

"I'm going to let you in on my little secret",

Elsa began.

@phenomnominal 2021

"You may have heard that any sufficiently advanced technology is indistinguishable from magic,"

"but actually, its the other way around..."

@phenomnominal 2021

"Any sufficiently incredible magic, is usually actually technology!"

@phenomnominal 2021

"Wow! That's amazing! I think?"

Kristoff was already blown away by this.

@phenomnominal 2021

"I'm just getting started!"

"To begin, we need to talk about data..."

said Elsa, with a grin.

@phenomnominal 2021

"The whole internet is full of data."

"Pictures"

"Memes"

"Infographics"

@phenomnominal 2021

{
  today: {
      temperatures: [19, 18.5, 18, 18.5, 17, /* ... */]
  },
  future: [
    { high: 21, low: 13, chanceOfSnow: 0.05 },
    { high: 21, low: 14, chanceOfSnow: 0.03 },
    { high: 21, low: 16, chanceOfSnow: 0.04 },
    { high: 19, low: 16, chanceOfSnow: 0.04 },
    /* ... */
  ]
}

"And that data is typically structured in some way, like this JSON  file"

@phenomnominal 2021

"Or like this HTML   file"

<h2>Weather result</h2>
<h3>
    <p>Arendelle, Norway</p>
    <p>Sunday 19:00</p>
    <span>Sunny</span>
</h3>
<img alt="Sunny" src="//arendelle.info/weather/64/sunny.png">
<h3>
  19<button aria-label="°Celsius">°C</button>
</h3>
<p>Precipitation: <span>0%</span></p>
<p>Humidity: <span>75%</span></p>
<p>Wind: <span>1 m/s</span></p>
<ol>
    <li>
        <p aria-label="Sunday">Sun</p>
        <img alt="Clear" src="//arendelle.info/weather/48/sunny.png">
        <span>21</span>°
        <span>13</span>°
    </li>
    <!-- ... -->
</ol>

@phenomnominal 2021

"Web developers use programming languages to inspect and manipulate that data"

const response = await fetch('https://arendelle.info/weather.json');
const data = await response.json();

// {
//   today: {
//       temperatures: [19, 18.5, 18, 18.5, 17, /* ... */]
//   },
//   future: [
//     { high: 21, low: 13, chanceOfSnow: 0.05 },
//     { high: 21, low: 14, chanceOfSnow: 0.03 },
//     { high: 21, low: 16, chanceOfSnow: 0.04 },
//     { high: 19, low: 16, chanceOfSnow: 0.04 },
//     /* ... */
//   ]
// }

const SATURDAY = 6;
const today = new Date().getDay();
const chanceOfSnowOnSaturday = data.future[SATURDAY - today].chanceOfSnow * 100;
h1 {
  // ...
}

h3 > p {
  // ... 
}

button:active,
button:hover {
  // ...
}

li:first-child {
  // ...
}

"Languages like JavaScript"

"or CSS"

@phenomnominal 2021

"Olaf, what exactly do you imagine when you think about snow?"

@phenomnominal 2021

"I think about skiing, and sledding, and snow fights!"

"It's so bright, and so cold, and so hard, but also so soft!"

"What do you think about Elsa?",

he asked.

@phenomnominal 2021

"I think about all those things too!"

"But my powers allow me to see another level deeper as well..."

"Going outside in the cold, having fun with our friends, ice skating..."

@phenomnominal 2021

"I see the whole system."

"In that system, each water molecule is a little bit of data, following the set of instructions that nature gave it."

@phenomnominal 2021

"Those instructions tell the water molecules when to freeze. "

"Those instructions tell the molecules which way to go when the wind blows."

"I know how to control those instructions."

@phenomnominal 2021

"l've worked it out!",

"You can understand the snow's source code!"

Anna announced suddenly.

@phenomnominal 2021

"Exactly!",

Esla grinned again.

"And do you want to know the best bit?"

@phenomnominal 2021

"The code that contains the instructions for all the snow in the world is written in JavaScript!"

"I can show you how to query code, modify code, and even create code from nothing!!"

"What do you think?"

@phenomnominal 2021

Chapter Three

Chapter THREE

The Snow Code

The Snow Code

Olaf was so excited that he could barely stop himself from bursting into song!

@phenomnominal 2021

"Will you teach us to use the snow code, Elsa?"

"Please, please, please, please, PLEASE!"

, he pleaded.

@phenomnominal 2021

"Of course I will!"

, Elsa reassured him.

@phenomnominal 2021

"But first, I need to teach you a few tricks!"

@phenomnominal 2021

"My gifts let me access the snow code directly!"

"You will need to use a slightly more direct approach..."

"Kristoff, have you heard of node.js?"

@phenomnominal 2021

"Yes!"

Kristoff looked pretty proud of himself.

"Anna taught me all about it."

@phenomnominal 2021

"Node.js is a computer program that lets us run JavaScript without a browser!"

@phenomnominal 2021

"We can use it to do things like read and write   files, find out stuff about the system, and manipulate data in lots of different ways!"

@phenomnominal 2021

"Exactly right! Let's start by looking at how we can read and write files using node.js"

@phenomnominal 2021

File System APIs

File System APIs

fs.watch

Node has functions that let us do all kinds of things with the file system:

  • you can check if you can access a file with
  • change file permissions with 
  • copy a file from one place to another with 
  • create a symlink with 
  • or set up a file watcher with 
fs.link
fs.copyFile
fs.chmod
fs.access

@phenomnominal 2021

File System APIs

File System APIs

The most important functions for us to learn today are:

fs.writeFile
fs.mkdir
fs.readFile
  •                                     - read a file from a path
  •                        - make a directory at a path
  •                                   - write a file to a path

@phenomnominal 2021

File System APIs

File System APIs

// snowflake.txt

                                ()
                                /\
                               //\\
                              <<  >>
                          ()   \\//   ()
                ()._____   /\   \\   /\   _____.()
                   \.--.\ //\\ //\\ //\\ /.--./
                    \\__\\/__\//__\//__\\/__//
                     '--/\\--//\--//\--/\\--'
                        \\\\///\\//\\\////
                    ()-= >>\\< <\\> >\\<< =-()
                        ////\\\//\\///\\\\
                     .--\\/--\//--/\\--/\\--.
                    //""/\\""//\""//\""//\""\\
                   /'--'/ \\// \\// \\// \'--'\
                 ()`"""`   \/   //   \/   `"""`()
                          ()   //\\   ()
                              <<  >>
                               \\//
                                \/
                                ()

Imagine that we have a file like this:

@phenomnominal 2021

import { promises as fs } from 'fs';

async function readSnowflake () {
  return fs.readFile('./snowflake.txt', 'utf8');
}

console.log(await readSnowflake());

File System APIs

File System APIs

We can use the Promise-based fs API to read the file, and print it:

@phenomnominal 2021

File System APIs

File System APIs

Which will give us something like this:

Weaseltown:dywbabt queenelsa$ node index.js

                              ()
                              /\
                             //\\
                            <<  >>
                        ()   \\//   ()
              ()._____   /\   \\   /\   _____.()
                 \.--.\ //\\ //\\ //\\ /.--./
                  \\__\\/__\//__\//__\\/__//
                   '--/\\--//\--//\--/\\--'
                      \\\\///\\//\\\////
                  ()-= >>\\< <\\> >\\<< =-()
                      ////\\\//\\///\\\\
                   .--\\/--\//--/\\--/\\--.
                  //""/\\""//\""//\""//\""\\
                 /'--'/ \\// \\// \\// \'--'\
               ()`"""`   \/   //   \/   `"""`()
                        ()   //\\   ()
                            <<  >>
                             \\//
                              \/
                              ()

@phenomnominal 2021

import { promises as fs } from 'fs';

async function readSnowflake () {
  return fs.readFile('./snowflake.txt', 'utf8');
}

async function saveSnowflake (data: string) {
  await fs.mkdir('./saved/snowflakes', { recursive: true });
  await fs.writeFile('./saved/snowflakes/snowflake.txt', data);
}

const snowflake = await readSnowflake();
await saveSnowflake(snowflake);

File System APIs

File System APIs

We can combine that with making a directory and writing the file to disk:

@phenomnominal 2021

Path APIs

Path APIs

It is also important to understand how the path API works:

path.resolve
  •                                       - resolve a path relative to the given path
  •                                 - split a path string into its constituent parts
path.parse

@phenomnominal 2021

import * as path from 'path';

const relativeToThisFile = path.resolve(__dirname, './snowflake.txt');
const parsed = path.parse(relativeToThisFile);

// interface ParsedPath {
//   root: string;
//   dir: string;
//   base: string;
//   ext: string;
//   name: string;
// }

Path APIs

Path APIs

import { promises as fs } from 'fs';
import * as path from 'path';

export async function readSnowflake () {
  let relativePath = path.resolve(__dirname, './snowflake.txt');
  relativePath = path.resolve(process.cwd(), './snowflake.txt');
  return fs.readFile(relativePath, 'utf8');
}

@phenomnominal 2021

, Sven snorted.

*snort*

@phenomnominal 2021

"That's right Sven! You must always test your file structure & path operations on all the different operating systems you care about! It's very easy to get it wrong!"

@phenomnominal 2021

"Now, we've seen how we can read a plain text file, what about something a bit different..."

@phenomnominal 2021

import { promises as fs } from 'fs';
import * as path from 'path';

async function readSnowflakes () {
  const snowflakePath = path.resolve(__dirname, './snowflakes.json');
  const json = await fs.readFile(snowflakePath, 'utf8');
  return JSON.parse(json);
}

console.log(await readSnowflakes());

We can read different types of data from different files:

JSON data

JSON data

@phenomnominal 2021

"Wow! So we took some data that was in a file, and parsed it into JavaScript objects that we can manipulate with code!"

Anna was really excited!

@phenomnominal 2021

"If you think that is cool, check this out..."

Elsa was just getting started!

@phenomnominal 2021

"If you think that is cool, check this out..."

const SNOW_CODE_PATH = path.resolve(
  __dirname,
  './elsa',
  process.env.SECRET_SNOW_FILE
);

Elsa was just getting started!

@phenomnominal 2021

"THE PATH TO THE SNOW CODE!"

Olaf instantly knew what it was!

const SNOW_CODE_PATH = path.resolve(
  __dirname,
  './elsa',
  process.env.SECRET_SNOW_FILE
);

@phenomnominal 2021

"Yes! The snow code is just another file!"

"It's just text and we can manipulate it and control it!"

@phenomnominal 2021

import { promises as fs } from 'fs';

export async function readSnowCode () {
  return fs.readFile(SNOW_CODE_PATH, 'utf8');
}

console.log(await readSnowCode());

Code as data

Code as data

We don't need to treat a code file any differently:

@phenomnominal 2021

Weaseltown:dywbabt queenelsa$ node secret_snow_code_file_dont_touch.js 

import { doPhysics, WaterMolecule } from './physics';

requestAnimationFrame(() => {
  for (let lat = -90; lat < 90; lat += 0.000000001) {
    for (let long = -180; long < 180; long += 0.000000001) {
      for (let alt = 0; alt < 80000; alt += 0.000000001) {
        const updated = update(lat, long, alt);
        render(updated);
      }
    }
  }
});

function update (lat, long, alt) {
  const molecule = global.waterMolecules[lat][long][alt];
  return doPhysics(molecule);
}

Code as data

Code as data

@phenomnominal 2021

"Now we have the snow code as a string, so  we can change it however we want!"

"This is amazing!"

Kristoff chimed in,

@phenomnominal 2021

// kristoff.js

import { promises as fs } from 'fs';
import { readSnowCode } from './read-snow-code';

const code = await readSnowCode();
  
code += `; console.log('Kristoff is the best!');`
  
await fs.writeFile('./kristoffs-cool-new-snow-code.js', code);

"I'm going to make it always  print out 'Kristoff is  the best!'"

, yelled Kristoff.

@phenomnominal 2021

// olaf.js

import { promises as fs } from 'fs';

const code = await fs.readFile('./kristoffs-cool-new-snow-code.js');

code = code.replace(/Kristoff/, 'Olaf');

await fs.writeFile('./kristoffs-cool-new-snow-code.js', code);

"Not if I can help it!"

, replied Olaf

@phenomnominal 2021

// sven.js

import { promises as fs } from 'fs';

const code = await fs.readFile('./kristoffs-cool-new-snow-code.js');

code = code.replace(/Olaf/, 'Sven');
code = code.replace(/best!/, 'bestest!!!!');
code = code.replace(/console.log\((.*)\)/, 'console.log($1.toUpperCase())');

await fs.writeFile('./kristoffs-cool-new-snow-code.js', code);

Even Sven had his own ideas...

@phenomnominal 2021

"Hold on everyone, l think we're getting ahead of ourselves!"

"Anna, can you think of any reasons why working with the string directly might not be the best approach?"

@phenomnominal 2021

"Strings aren't exactly structured!"

"You're just changing a bunch of characters, and that could break easily!"

"Regular Expressions are hard to get right, and hard to understand later!"

"You have to fight with comments, and whitespace!"

@phenomnominal 2021

"So how can we get the snow code into a more useful structure?"

@phenomnominal 2021

import * as path from 'path';

const parsed = path.parse('/some/path/to/some/file.txt');

// interface ParsedPath {
//   root: string;
//   dir: string;
//   base: string;
//   ext: string;
//   name: string;
// }

console.log(parsed.ext);
// read-snowflakes.js

import { promises as fs } from 'fs';
import * as path from 'path';

export async function readSnowflakes () {
  const snowflakePath = path.resolve(__dirname, './snowflakes.json');
  const data = await fs.readFile(snowflakePath, 'utf8');
  const snowflakes = JSON.parse(data);
  
  console.log(snowFlakes[0].velocity.x);
}

Parsing

Parsing

Let's look back at how we handled the other strings:

@phenomnominal 2021

import { parseScript } from 'esprima';
import { readSnowCode } from './read-snow-code';

const code = await readSnowCode();
const ast = parseScript(code);
console.log(ast);

Parsing

Parsing

We also need to parse our string of code!

@phenomnominal 2021

"An ast, I LOVE IT!"

"What is an ast?"

@phenomnominal 2021

"Good try Olaf..."

"It's actually an A S T "

"It's an Abstract Syntax Tree..."

@phenomnominal 2021

Chapter Four

Chapter Four

The root of all problems

The root of all problems

"Oh yeah of course, I know all about those kinds of trees..."

"But I think Sven needs a bit of a refresher..."

Kristoff sounded uncertain.

@phenomnominal 2021

"Sure! Let's start with a literal meaning."

@phenomnominal 2021

Abstract Syntax Trees

Tree

data structure made up of vertices and edges without any cycles

SYNTAX

the way in which linguistic elements are put together

Abstract

Not associated with any specific instance

Abstract Syntax Trees

@phenomnominal 2021

????

????

????

????

????

????

????

????

????

@phenomnominal 2021

Abstract Syntax Trees

An Abstract Syntax Tree is a data structure that represents the structure of code, but without any actual syntax.

let it = go('let it go');
it = go 'let it go'
it <= go ['let it go']
'let it go' -> go => it

Abstract Syntax Trees

@phenomnominal 2021

{
  type: 'Program',
  body: [{
    type: 'VariableDeclaration',
    declarations: [{
      type: 'VariableDeclarator',
      id: {
        type: 'Identifier',
        name: 'it'
      },
      init: {
        type: 'CallExpression',
        callee: {
          type: 'Identifier',
          name: 'go'
        },
        arguments: [{
          type: Literal',
          value: 'let it go'
        }]
      }
    }],
    kind: 'let'
  }]
}

@phenomnominal 2021

            'let it go'

Literal

Identifier

CallExpression

Identifier

VariableDeclarator

VariableDeclaration

         go
         go('let it go')
    it
    it = go('let it go')
let it = go('let it go');

@phenomnominal 2021

Identifier

IfStatement

ImportDeclaration

Literal

CallExpression

"Learning these abstract names and this new data structure is tricky, but there are tools to help!"

ClassDeclaration

FunctionDeclaration

VariableDeclaration

ExpressionStatement

AwaitExpression

ForStatement

WhileStatement

DoWhileStatement

@phenomnominal 2021

"AST Explorer lets you look at the parser output for a huge number of languages!"

"It is very useful for seeing the structure of the tree and what properties are present!"

AST Explorer

AST Explorer

@phenomnominal 2021

@phenomnominal 2021

https://astexplorer.net/

import { parseScript } from 'esprima';

const code = `let it = go('let it go');`
const ast = parseScript(code);

const arg = ast.body[0].declarations[0].init.arguments[0];

console.log(arg);

"Seeing the whole tree helps to write out long complex property access chains"

@phenomnominal 2021

"So Anna, what do you think? Is this better than the string version?"

@phenomnominal 2021

"l think so!"

"We have a real structure now with the AST"

"We access data exactly like we would with JSON, so it's easier to understand"

"We don't have to think about comments, or whitespace, we just access what we want to!"

@phenomnominal 2021

"lt's still kind of clunky though..."

"Do you think we can do better?",

"And it will throw an error if the structure doesn't line up with the code!"

Elsa asked.

@phenomnominal 2021

{
  "kind": "document",
  "children": [{
    "kind": "h2",
    "children": [{
      "kind": "text",
      "text": "Weather result"
    }]
  }, {
    "kind": "h3",
    "children": [{
      "kind": "p",
      "children": [{
        "kind": "text",
        "text": "Arendelle, Norway"
      }]
    }, {
      "kind": "p",
      "children": [{
        "kind": "text",
        "text": "Sunday 19:00"
      }]
    }, {
      "kind": "span",
      "children": [{
        "kind": "text",
        "text": "Sunny"
      }]
    }]
  }, {
    "kind": "img",
    "alt": "Sunny",
    "src": "//arendelle.info/weather/64/sunny.png"
  }]
}

"Let's look at another data structure..."

@phenomnominal 2021

"Olaf, what do you think this data structure represents?"

@phenomnominal 2021

{
  "kind": "document",
  "children": [{
    "kind": "h2",
    "children": [{
      "kind": "text",
      "text": "Weather result"
    }]
  }, {
    "kind": "h3",
    "children": [{
      "kind": "p",
      "children": [{
        "kind": "text",
        "text": "Arendelle, Norway"
      }]
    }, {
      "kind": "p",
      "children": [{
        "kind": "text",
        "text": "Sunday 19:00"
      }]
    }, {
      "kind": "span",
      "children": [{
        "kind": "text",
        "text": "Sunny"
      }]
    }]
  }, {
    "kind": "img",
    "alt": "Sunny",
    "src": "//arendelle.info/weather/64/sunny.png"
  }]
}

@phenomnominal 2021

"IT LOOKS LIKE HTML!"

@phenomnominal 2021

"That's an approximation of what the browser creates when it downloads an HTML    file and parses it!"

"Very good!"

"We call it the Document Object Model, or the DOM."

"It is also a tree!"

@phenomnominal 2021

"Now, wouldn't it be strange if we queried the DOM  like this?"

import { parse } from './parse-html';

const HTML = `
    <h3>...</h3>
    ...
`;

const dom = parse(HTML);

const node = dom.children[1].children[3].children[0];

console.log(node);

@phenomnominal 2021

"It's much more likely that we would do something like this:"

const dom = parse(HTML);

const node = $(dom, 'body > h3 > span:last-child');

console.log(node);

"CSS selectors are a very powerful way to navigate a tree!"

@phenomnominal 2021

"So why don't we just use CSS selectors with the JavaScript AST?"

"Does JavaScript let us do that?"

@phenomnominal 2021

"Unfortunately no, not out of the box..."

"Fortunately, there's a huge JavaScript open-source community, and some
wonderful people made
it so we can do this!

@phenomnominal 2021

@phenomnominal 2021

import { parseScript } from 'esprima';
import { query } from 'esquery';

const code = `let it = go('let it go');`;
const ast = parseScript(code);
const query = 'CallExpression:has(Identifier[name="go"]) > Literal';
const nodes = query(ast, query);

console.log(nodes);

"The query helper means that the chained property access vanishes!"

ESQuery

ESQuery

@phenomnominal 2021

"Woooow"

Olaf was impressed

@phenomnominal 2021

"I get it! These tools are amazing!"

"We can find out if a file has an identifier with a certain name!"

"Or all the names of all the exported functions!"

"Or if it matches some syntactical pattern!"

@phenomnominal 2021

"Like a lint rule!?"

"Yes! AST queries are *perfect* for writing lint rules!"

@phenomnominal 2021

"We can write a custom lint rule in only a few lines!"

import { parseScript } from 'esprima';
import { query } from 'esquery';

const code = `Object.freeze()`;
const ast = parseScript(code);
const query = 
  'CallExpression:has(MemberExpression[object.name="Object"][property.name="freeze"])';
const nodes = query(ast, query);

if (nodes.length !== 0) {
  throw new Error(`Don't use Object.freeze!`);
}

@phenomnominal 2021

"But wait. What do these queries have to do with the snow code?"

@phenomnominal 2021

"In order to change how the snow code works, we need to be able to add our code in the right place!"

"This is where the real magic begins..."

@phenomnominal 2021

Chapter Five

Chapter Five

Real Magic

Real Magic

"My magic works by changing how physics works in real-time."

"Let's look at the snow code again..."

@phenomnominal 2021

import { doPhysics } from './physics';

requestAnimationFrame(() => {
  for (let lat = -90; lat < 90; lat += 0.000000001) {
    for (let long = -180; long < 180; long += 0.000000001) {
      for (let alt = 0; alt < 80000; alt += 0.000000001) {
        const updated = update(lat, long, alt);
        render(updated);
      }
    }
  }
});

function update (lat, long, alt) {
  const molecule = global.waterMolecules[lat][long][alt];
  return doPhysics(molecule);
}

"We need to wrap the "doPhysics" function call with some magic!"

@phenomnominal 2021

import { parseScript } from 'esprima';
import { query } from 'esquery';
import { readSnowCode } from './read-snow-code';

const PHYSICS_CALL_QUERY = 'CallExpression:has(Identifier[name="doPhysics"])';

const snowCode = await readSnowCode();
const ast = parseScript(snowCode);
const doPhysicsCalls = query(ast, PHYSICS_CALL_QUERY);

doPhysicsCalls.forEach(doPhysicsCall => {
  doPhysicsCall.expression.name = 'doMagic';
  const doMagicCall = doPhysicsCall;
  doMagicCall.arguments = [{
    type: 'CallExpression',
    callee: {
      type: 'Identifier',
      name: 'doPhysics'
    },
    arguments: [{
      type: 'Identifier',
      name: 'molecule'
    }]
  }];
});

"We can do that like this"

@phenomnominal 2021

import { doPhysics, WaterMolecule } from './physics';

requestAnimationFrame(() => {
  for (let lat = -90; lat < 90; lat += 0.000000001) {
    for (let long = -180; long < 180; long += 0.000000001) {
      for (let alt = 0; alt < 80000; alt += 0.000000001) {
        const updated = update(lat, long, alt);
        render(updated);
      }
    }
  }
});

function update (lat, long, alt) {
  const molecule = global.waterMolecules[lat][long][alt];
  return doMagic(doPhysics(molecule));
}

"That gives us something like this"

"Can you spot the problem, Kristoff?"

@phenomnominal 2021

"We didn't create the "doMagic" function  yet!"

"I think that it  should freeze all the molecules!"

@phenomnominal 2021

"That's right! And what a great idea!"

"Generating code is pretty verbose when you manually create AST tokens..."

@phenomnominal 2021

// function doMagic (molecule) {
//   molecule.setTemperature(-10);
//   return molecule;
// }

function createMagicFunction (ast) {
  return {
    type: "FunctionDeclaration",
    id: {
      type: "Identifier",
      name: "doMagic"
    },
    params: [{
      type: "Identifier",
      name: "molecule",
    }],
    body: {
      type: "BlockStatement",
      body: [{
        type: "ExpressionStatement",
        expression: {
          type: "CallExpression",
          callee: {
            type: "MemberExpression",
            object: {
              type: "Identifier",
              name: "molecule"
            },
            property: {
              type: "Identifier",
              name: "setTemperature",
            },
          },
          arguments: [{
            type: "UnaryExpression",
            operator: "-",
            argument: {
              type: "Literal",
              value: 10
            }
          }]
      }
    }, {
    type: "ReturnStatement",
      argument: {
        type: "Identifier",
        name: "molecule"
      }
    }]
  }
}

"But it would look something like this..."

@phenomnominal 2021

"Yeeesh"

"You bet we can! We've one last trick up our sleeves!"

"We can do better than that can't we Elsa?"

@phenomnominal 2021

const h2 = document.createElement('h2');
h2.innerText = 'Weather result';
const h3 = document.createElement('h3');
const p1 = document.createElement('p');
p1.innerText = 'Arendelle, Norway';
const p2 = document.createElement('p');
p2.innerText = 'Sunday 19:00';
const span = document.createElement('span');
span.innerText = 'Sunny';
h3.appendChild(p1);
h3.appendChild(p2);
h3.appendChild(span);
const img = document.createElement('img');
img.setAttribute('alt', 'Sunny');
img.setAttribute('src', '//arendelle.info/weather/64/sunny.png');

"Let's go back to the DOM "

"Wouldn't it be quite odd to create elements like this?"

@phenomnominal 2021

const template = `
  <h2>Weather result</h2>
  <h3>
    <p>{{ location }}</p>
    <p>{{ time }}</p>
  <span>{{ weather }}</span>
  </h3>
  <img
    alt="{{ weather }}"
    src="//arendelle.info/weather/64/{{ weather }}.png">
`;

render(template, {
  location: 'Arendelle, Norway',
  time: 'Sunday 19:00',
  weather: 'Sunny'
});

"We're more likely to do something like this!"

@phenomnominal 2021

"That'd be pretty amazing! Does something like that exist for JavaScript?"

*clip clop*

@phenomnominal 2021

"It sure does!"

@phenomnominal 2021

@phenomnominal 2021

The template helper means that all that code squashes down to just this:

import { query } from 'esquery';
import { compile } from 'estemplate';

const DO_MAGIC_TEMPLATE = tstemplate.compile(`
  function doMagic (molecule) {
    molecule.setTemperature(-<%= temperature %>);
    return molecule;
  }
`);

function createMagicFunction () {
  const doMagicAst = DO_MAGIC_TEMPLATE({
      temperature: createNumericLiteral('10')
  });
  const [doMagicFunction] = query(doMagicAst, 'FunctionDeclaration');
  return doMagicFunction;
}

ESTemplate

ESTemplate

@phenomnominal 2021

"That is SO MUCH BETTER!!"

@phenomnominal 2021

"It sure is!"

"We have much less code to maintain!"

"Doing AST-based templating like this also makes it much easier to transplant and clone parts of the tree around!

@phenomnominal 2021

"Now all we need to do is convert the AST back into text!"

"For that, we get to use one more open-source library!"

@phenomnominal 2021

@phenomnominal 2021

The generate helper takes a valid AST and turns it into JavaScript!

import { parseScript } from 'esprima';
import { query } from 'esquery';
import { generate } from 'escodegen';

const code = `let it = go('let it go');`;
const ast = parseScript(code);
const query = 'CallExpression:has(Identifier[name="go"]) > Literal';
const [literal] = query(ast, query);
literal.value = 'LET IT GO!';
const updatedCode = generate(ast);

console.log(updatedCode); // let it = go('LET IT GO!');

ESCodeGen

ESCodeGen

@phenomnominal 2021

"Now everyone, we're ready to put all the bits together!"

"I'm going to need all your help!"

@phenomnominal 2021

import { readSnowCode } from './read-snow-code';

const snowCode = await readSnowCode();

import { parseScript } from 'esprima';

const ast = parseScript(snowCode);

import { query } from 'esquery';

const doPhysicsCalls = query(ast, 
  'CallExpression:has(Identifier[name="doPhysics"])'
);

import { template } from 'estemplate';

doPhysicsCalls.forEach(oldDoPhysicsCall => {
  oldDoPhysicsCall.expression.name = 'doMagic';
  const doMagicCall = oldDoPhysicsCall;
  const newDoPhysicsAst = template('doPhysics(molecule)');
  const [newDoPhysicsCall] = query(newDoPhysicsAst, 'CallExpression');
  doMagicCall.arguments = [newDoPhysicsCall];
});
  
const doMagicAst = template(`
  function doMagic (molecule) {
    molecule.setTemperature(-<%= temperature %>);
    return molecule;
  }
`, {
    temperature: createNumericLiteral('10')
});

const [doMagicFunction] = query(doMagicAst, 'FunctionDeclaration');

ast.statements.push(doMagicFunction);

import { generate } from 'escodegen';

const updatedSnowCode = generate(ast);

import { writeSnowCode } from './write-snow-code';

await writeSnowCode(updatedSnowCode);

"Are you ready for this?"

@phenomnominal 2021

"We just did a lot of things!"

  • Read the "Snow Code" with the "fs" and "path" APIs
  • Use ESPrima to convert the code into an AST
  • Use ESQuery to query the AST for the "doPhysics" CallExpression
  • Modify the AST to pass the result of "doPhysics" to "doMagic"
  • Use ESTemplate to create the code for the "doMagic" function
  • Add the new FunctionDeclaration back to the AST
  • Use ESCodeGen to print the AST back to a string
  • Save it to disk

"Let's recap!"

@phenomnominal 2021

"That was a lot of information"

@phenomnominal 2021

"These ideas are the foundation of all tools that help build the internet!"

"From Webpack to ESLint, from Sass to Create React App"

"And now we can make our own!!"

@phenomnominal 2021

"Remember you can use these ideas to modify any source code in any programming language!"

@phenomnominal 2021

"Elsa, thank you so much for showing us your magic!"

, said "Sven".

@phenomnominal 2021

"Don't you mean 'technology'?"

Everyone laughed, and they started heading back towards Arendelle.

, said Anna.

@phenomnominal 2021

The End

The End

Okay, Bye

Okay, Bye

  • twitter.com/phenomnominal
  • github.com/phenomnomnominal
  • linkedin.com/in/craig-spence

Do You Want to Build a Build Tool? - Infoshare 2021

By Craig Spence

Do You Want to Build a Build Tool? - Infoshare 2021

Infoshare 2021

  • 3,263

More from Craig Spence