The Different Transformation Stages of AOT Compilers by Eytan Manor | JSConf.Asia 2019

Subtitles section Play video

Ah, so Hey, guys, I hope you're having a great time in Singapore.
I love Singapore personally.
Are there any people from Israel, by the way, can raise their hand.
Okay, I guess not.
So Ah, First of all, let me introduce myself.
My name is eight on I come from Israel.
At the moment I live in Taipei.
I'm a full stack JavaScript engineer.
I do that full time and I also do that for fun.
This is why I give a talk right now.
And when it comes to programming, I really like to dive deeply into stuff which are trivial, so trivial that it's part of our daily routine and we just overlook it, Um, and because it's something so trivial, then I'm gonna dive deeply into it, and naturally, it's gonna be a little of a level talk, so it's not gonna turn you necessarily into better react or angular developers.
But at the end of the day, you'll be able to solve problems that a lot of programmers don't know how to solve, and just a new world of possibilities will open up to you.
And the subject of the day is a ot compilers ahead of time.
So I assume that you're pretty much familiar with the concept.
Okay.
Example compilers compilers for C or C plus Plus, where you basically take a coat string and you get it from one formats into native machine code.
Or, in the case of javascript, where you take yes, next and you compel it to Yes, five.
No.
Why is it such a trivial topic?
To prove my point, I have a few questions.
If it's true for you, please raise your hand.
Who here uses Babel to compile jumps with applications?
Please raise your hand.
Okay.
A lot of people who here uses typescript raise your hand.
Okay again.
A lot of people.
Now listen carefully.
This is slightly different.
Who here uses Yes, five to write jealousy applications.
Please raise your hand.
You get out.
Get out!
No.
Okay, so we see that there's a common pattern here, right?
We all use Babel and typescript and all these sort of compilers.
But no one ever stopped and wonder how does it actually work?
And there's a lot of advantages to that, because once you know how it actually works, then you would know how to run some processing before you actually go to the interpreter rather than race wasting some runtime processing power.
So I'm gonna go through very trivial stuff.
Um, and later on, I'm gonna dive deeply into that, So just bear with me.
Um, yeah.
So why exactly do we use ahead of time compilers?
What are the advantages?
First of all, you can use it to optimize coat.
So let's say that we have parts of God that we don't actually use, such as tree shaking, wary import modules, and we end up not using these models.
Therefore, we can you reduce that part of goat and reduce bundle size.
Or, for example, if we have variable assignments off variables that we don't actually use.
Then we can just delete that when we compel the code and therefore safe some memory locations.
And one night what night?
Why not take it a step further?
Using winters eso basically what the lint or does it says?
Hey, man, listen, there's something wrong here, okay?
You import that module and you don't use it.
Or maybe there's a standard of coat that we want to just live up to our own expectations.
So let's say we want to use ah conditions in a specific way.
Maybe closures in specific places maybe will want to use to force use constant variables.
So before we actually run the optimization itself, we can basically make the code look a lot better before we push it into our, um, get host.
Also, it enables backward compatibility.
Now, we all, um, use the latest features of JavaScript and evil.
Just open grown deaf tools right now.
And I'll try to use, um, syntax.
Such a zero functions, a sink, a weight and so on.
It will work.
But we all know that unfortunately, there are Internet Explorer 11 out there.
Okay, uses that use this old browser.
And we need to enable backward compatibility because we don't know what type of browser, what version of the browser.
The user's gonna run.
Therefore, we need to ensure that we use the least amount of features possible when it comes to Emma script.
Also, when someone suggests the new feature for the set of specifications for a script, it takes time before it actually gets accepted into the language.
Because the comedian it's to look at it afterwards, the community needs to also see how it fits within the eco system.
And it's to go through stages 0123 and, uh, then all the browsers they need to implement that natively within c++, and they all need to be aligned with one another.
This process takes time using an external compel.
Er, this is relatively easy to just plug in and other feature, and then basically there's a layer on top, which is common for all browsers.
So it's very easy, especially in the case of Babel, where the compiler is written in a high level language such as javascript, and we also have the opportunity to test the new feature and see if it actually fits.
One final reason is that, um, we can use costumes, syntax also no known as language extensions such as typescript, where we basically have type safety.
Okay.
And we can see if the code actually behaves the way we want.
Or we can use Jace X, which is used vastly too, right?
React applications in a better way.
So obviously there are advantages to the compilers.
Wait.
Okay.
Um, so there aren't gonna be many slides.
This there's gonna be this light and another one and that's it is mostly gonna be demonstration.
I hope I can get it right, because I'm not used to the set up.
Ah, like they said, this is like I mean, I'm used to lean ox, and this is like another world for me.
Um, anyways ah, what are the stages off the copulation?
Everything that I'm gonna mention?
It's, um it is true for all ot compilers, I believe.
Although I want to be more specific to Babel because it's very popular within the eco system.
It's also very easy to write blankets into Babel.
Ah, but you everything is true for all.
Um 80 compel is out there.
So first stage, first of all we take is an input, the code string, and then we parse it parsing.
What it means is that we put it into a chase on.
Okay, This Jason is basically a tree, and the tree is made out of notes.
Each and every note without in battery represent a different feature within the syntax.
This tree is called a s t absolute Syntex tree.
Because it represents the syntax.
No, comes the second stage.
Once we have the ASD we take it is an input and we generate code out of it in a different format.
Okay, it's very easy to do so we basically take a specific note and then we generate the magic coat strings to that specific note In the case off Babel, it's it's slightly different, but because Babel is it's not exactly a compelling, but it works exactly the same way.
It's more like a transformer.
So even though you take an SD, the generated coat should be exactly the same.
Now I put a lot of information out there and stuff like that.
Um, the best way to explain something is by demonstrating.
So this is exactly what I'm gonna do.
No.
So let's Okay, so let's focus on the A S t first.
How does the ASD looks like?
Exactly.
So there's a website out there.
It's called a sea Explorer.
This one and well, it's okay.
And we have two windows in here.
Okay.
In the left window.
What we have is a text editor where we can type code in the right.
We have an object, the jumpsuit object.
As I said, the ASD that represents the code so What happens if I They've just a very generic code.
Nothing special about it.
So we see on the right that the view gun updated.
Okay.
And on the right, this actually, this is the ASD that represents the code.
Now, what's nice about this stool is not is that not only it shows us the ASD, but it's very easy to highlight the code and match it with the correlated note.
So let's highlight something on the left.
Okay?
Wait.
He doesn't use the right parts, sir.
Okay.
Also explain what it is.
So once I highlight something on the left, it will also be highlighted on the right and vice versa.
If I hover over a specific no type, it will be highlighted on the left.
So it's very easy to see what represents the code.
And if we'll take a specific note is an example, let's highlight number one over there, and we'll look at the note.
We'll see that the type name actually matches the description.
Okay, this is called a numeric literal, because literally, this is a number with the value one.
So this is basically a stand.
You see their nose which are nested with with one another and it all starts from the root.
So this principle is true for all syntax and languages and a tree doesn't need to look a certain way they there can be many partners.
So, baby long seven is what ah Babel used to use under the hood.
Now they renamed it.
They made it part of Babel's Monory Po and they call it just Babel part sir.
Ah, but there are other parts is such as Acorn s prima Ah which played Ah ah!
Very important role in when it comes to partners in JavaScript.
Um, flow and typescript and not only Java script.
There can be many languages.
There can be C s s graphical html handlebars.
Travis, these principles apply to all syntax and languages.
Okay?
No.
How exactly does it look within a know Js application?
How exactly do I take code and parts it into an HST?
So because I'm here right now, I can just say anything and you would believe me, but I don't want that.
Okay?
Everything that I say I want to back it up with proofs.
So for that we will go to Aa Babel's documentation website Okay, Okay.
And in the search wire, there's a library called Babel.
Parsa.
I hope you can see that.
Yeah, and the A p I is fairly simple.
We just import the parts function and we provided with the coat string and the output should be an ASD.
Now it accepts all different kind of options.
But there there is.
I don't want to go go through it because I want to keep things simple.
But however, there is a specific option that would like to focus on this is the plug ins option.
By default, it will parse only a limited set off features off notes.
But there are different notes which are not officially part of magma script notes which are in stage 012 or three so they won't be parsed by default.
And for that, we need to say to the to the part, sir, explicitly that it needs to parse them so features that you're probably familiar with, such as a single aerator big in that we just talked about glass property is private class properties and thinks that I'm quite frankly not even formula with, um And we can also tell the Parsa to support no types of language extensions.
Okay, So like I mentioned earlier there some features which are not part of any specification such as flow just x so under the hood, this is how it happens.
Babel officially supported and also typescript.
That's interesting.
This is not what typescript uses.
Typescript has a dedicated Parsa and compiler.
Although Babel also supports parsing typescript limited features of of it, it's still nice to have.
So now let's see a demonstration in the notorious application.
Yeah.
So this is a very simple program.
What I have here, I import the parts function and then I provided with a coat string, and then I parse it, and the output is an ASD.
And what I do, I print to the council a string ified version off the ASD and the expected Albert should simply be an SD.
Now I'll run the program.
Wait again.
This is an environment I'm not really used to.
Okay, so the alternative is on a stick.
This is just the Jason Jason that this describe describes the code.
I'm not gonna go through it because this is very boring, but I think we got the principal once we saw because of what we saw on Honesty Explorer.
Now the second stage.
Once we have a nasty we put it back into code.
This stage is called generating, so I'll go to Babel's documentation website again and we will look at the library called Generator.
So again, very simple.
A p I we simply accept NASD as the first argument the output should be coat and again I would like to demonstrate that very quickly.
Okay, good.
So in addition to what I did earlier No, I have the generate function and once I import it, I provided with a generated A S T and I printed to the console and the output should be exactly the same.
And let's run the program again.
You see, this is it's not exactly the same.
It's slightly different because there are semi colons, why they're semi colons.
Because the ASD doesn't contain information about the semi colons, it simply puts the coat back in the way it can.
And no, I want to go back to the slides.
By the way, you see that the output is exactly similar to the input.
Now, when we look at the traditional compiler the input and the output should be different.
Okay, when?
When you look at simple spots, for example, because the output is a native machine code.
But Babel is not exactly a compiler.
It's like a transformer or trans filer, and it works in a slightly different way.
Um, but soon I'll talk about it and we'll see how from.
Yes.
Next it goes to E s five.
So back to the slides, I would like to add another optional stage.
It comes between stage one and stage two.
Okay, So stage to become Stage three.
And why this is like I do it like that.
This is an optional stage because most compilers Sorry, a compiler doesn't necessarily need to have that stage in order to work properly.
But this is a very powerful stage, because using that stage we can act actually apply pre processing to the code and run.
Optimization is before we actually address the interpreter.
And this is by far the most complex stage because this is a P I specific.
There's a lot, a lot of things to know about the A p I, and will soon see why.
So going back to the code?
No, let's say that it would like to transform the code with traditional methods using only regular expression.
This would be very complicated because a code can have scopes.
And maybe there's, ah, a piece of code which is wrapped with brackets or something like that, which makes the, um, transformation irrelevant.
This is why we have the ASD.
The ASD makes it easy on us to transform the code.
So let's say that I would like through transformation to change const see into constant be.
So how do I do that?
What do I need to do is to address the right note with the right property and then change its value.
And once we put it back into a coterie because the coat strings the new culturing is derived from the A S t.
Then we should have a transformed coat.
So we copy that and we go to a hasty export.
This is how I actually work when I write Babel plug ins or I I want to transform code and I placed it okay, and then we see that it got updated on the right.
Now what I do, I highlight the code that I would like to transform.
And then the corresponding note is being highlighted on the right.
Now what?
I d'oh, I need to look for the path that represents the note.
So I start from the root and I try I write down the path so it goes to program Body position zero declarations position zero I d.
Name.
Let's say that I wrote everything down in the text editor, and I have it right here.
Yes.
Yeah.
So I changed the ASD.
I changed the values see to be.
And therefore, once I put it back into code, the expected output should be the following.
You see?
So I changed the code.
No, there's a problem with that approach because what I did right now is too specific.
I want to have, like, some sort of a generic pattern that I can apply.
And it will be true for any code string that I get as an input, something like regular expression.
Okay, because a code that can wear many shapes and forms, this is way too specific.
So there's a problem with that approach.
Many compilers they implement a mechanism and they provided provide us with an A P I, which lets us transformed the code in a more generic way.
Okay, so using that a p I what we do, we basically walk through the a S t through each and every note in the A C.
You can call it walk or run or traverse essentially these verbs.
They all represent the same thing.
And then we apply transformation callbacks for each and every note individually.
So if we look at the ASD and we'll try to demonstrate it, what he does, it goes to It starts with the route.
It goes to the root knowed it goes deeper to program, verbal the collision and then once it sees a transformation cold back then it goes to the callback, applies the logic and then it keeps walking through the tree.
If we look at Babel, Babel's documentation website, there's a library called Babel Traverse and Traverse does.
Exactly what I said is the first argument.
It accepts NASD as the second argument.
It accepts an object which is called a visitor, because this object visits the ASD as we walk through it, and this object contains callbacks.
The names of the callbacks represents the notes that is it is going to transform.
And as the first argument it accepts an object that contains the note.
And then we can actually do all over changes on the know that we would like to transform, and we can also apply conditions and logic to it.
It also accepts more arguments.
I don't want to get into that because I want to keep things very simple.
Once you understand the core, it's very easy to get deeper and investigate more about the subject.
So let me show you another demonstration of that.
So in addition to what I had earlier, no, I used it Reverse function to traverse through the A S t.
And I apply it transformation callbacks.
So if I have that input, what it would like to have hope, the expected output should be something like that.
So let's say that I change all the variable assignments from whatever this is to be.
Everything should be be and the values should all change to to and f n She changed to F in two.
So this is the expected output.
So how do I do that?
I use the right call back names with the right visitor And then I change the corresponding notes again.
How do I know?
How does the AP I looks like and what notes to address?
I go to a s, the explorer, and then I highlight the code, the the parts of the code that I would actually like to transform.
And then I look at the node.
I see.
What is the name off the node?
And I see how the signature of the nodes off the note looks like.
And this is how I know how the AP I looks like No, if only run the code, the output should be what I just wrote over there, you see.
And just to prove it to you, that this is a general pattern and it's not just specific to that code.
I can add another variable assignment over here.
And this change to be equals two as well, right?
This is nice.
No, this is how we transform code.
So the visitor over here, this is what a Babel plug in is all about.
A plug in is just a visitor, Okay?
Mabel wanted to make it easy on us and they wanted to give us a way to implement custom features using plug ins.
And so, in order to do that when we need to do, is simply to write a visitor.
And what we do we specified in the Babel RC than it looks for the right name into not modules.
And then it applies.
The visitor, once it walks through the ASD before it actually compiles it.
No, let's see an example plug in.
So what I have here is a program which the input is exactly the same as we had earlier.
And the plug in is exactly what we had earlier.
Again, a plugging is a visitor.
It accepts more options that will make it easy on us to, ah, transform the code.
But again, I want to keep things very simple because the visitor is the same and the input is the same.
Therefore, the output should be exactly the same as well.
So I'm gonna run the program again.
And the output is exactly the same.
No.
How exactly from that Do we rich?
Yes.
Five.
Okay, because so far it just it looks like it remains to be yes, next.
So, yes, five is simply a set of plug ins that transforms the code in a way that it will from Yes, next, it will turn into us five.
A set of blogging is also called a preset.
This is what a prison is.
A preset is simply a set of plug ins.
So in order to transform to us five what we do, we load a set off plug ins using the Babel envy.
Pre sent the envy, envy, preset.
What?
It does it apply ice plug ins based on the environment that we would like to transform too.
Okay.
How does it know that based on what's written on Pecker, Jason, No.
I'm going to apply this plug in and you'll see that will be transformed to us five.
And let's make the input a little bit more.
Ah, yes.
Next.
And we'll run the demo again.
And the output is this Five, as you can see.
So this is how Babel works.
Okay?
This is how it works Now.
Um, what's the point of it?
Okay.
Why is it such a powerful tool?
Because using that knowledge, you can write plug ins yourself and enhance the user experience as a programmer.
So I'll give you example plug ins that I wrote myself.
You don't necessarily have to agree with what they do.
Maybe you don't like their functionality, but again, just understand the power behind that sort of information.
Okay, so first blogging is a plugging for Ah, if you're from in there with the new react hooks a p I here what I do, I transform data, Okay?
And the problem is that once I entered the rendering face each and every time, I would need to apply that logic or he here.
If I have anonymous call back each time I enter the rendering phase, I would need to initialize that callback.
So what I need to do is to cash the result using your use call back or use memo.
So with that plug in, you can automatically apply Yoo's memo with the right arguments.
You just type it normally as you would, and then it optimizes the coat for you ahead of time.
Or he also does things like creating anonymous components where it cashes the callback results and you don't need to put use cold back on.
Therefore, safe some errors second and final, plugging that I wrote something for style components rather than creating a dedicated component pair each component that you would like to encapsulated style.
You can use a single style shit component and use underscore to encapsulate private class names.
And what the plug in will do It will perfect sit automatically.
Therefore encapsulate the styles.
So, um, that was basically it.
I hope you liked it.
Ah, there isn't much information about it.
Online documentation is very, very poor.
Ah, but I did write few articles about it, and I'm also gonna upload a video about it.
And also be sure to use a stick Spore and Babel docks.
If you have any further questions, feel free to address me later on in the break.
Andi.
Thank you.