@samitouri / QOS-React-2 / commits / 3e317b8bbf

[rust] Contributing guide and readmes for every crate

Per the title, this updates the main readme file with a guide to contributing to the Rust compiler, and ensures that we have a brief description of every crate in local readme.md files. The `forget_hir` one is the most extensive and describes the high-level design of the HIR.

Joe Savona committed Jul 14, 2023 at 12:24 UTC 3e317b8bbfe4d3861730c5739a76670903bfc97b
8 files changed +143 -2
compiler/forget/README.md
+78
@@ -33,3 +33,81 @@ Scaffolding
33 Reference
34
35 - [Babel Plugin Handbook](https://github.com/jamiebuilds/babel-handbook/blob/master/translations/en/plugin-handbook.md)
36 +
37 +## Rust Development
38 +
39 +## First-Time Setup
40 +
41 +1. Install Rust using `rustup`. See the guide at https://www.rust-lang.org/tools/install.
42 +2. Install Visual Studio Code from https://code.visualstudio.com/.
43 + Note to Meta employees: install the stock version from that website, not the pre-installed version.
44 +3. Install the Rust Analyzer VSCode extension through the VSCode marketplace. See instructions at https://rust-analyzer.github.io/manual.html#vs-code.
45 +4. Install `cargo edit` which extends cargo with commands to manage dependencies. See https://github.com/killercup/cargo-edit#installation
46 +5. Install `cargo insta` which extens cargo with a command to manage snapshots. See https://insta.rs/docs/cli/
47 +
48 +## Workspace Hygiene
49 +
50 +### Adding Dependencies
51 +
52 +To add a dependency, add it to the top-level `Cargo.toml`
53 +
54 +```
55 +// forget/Cargo.toml
56 +[workspace.dependencies]
57 +...
58 +new_dep = { version = "x.y.z" }
59 +...
60 +```
61 +
62 +Then reference it from your crate as follows:
63 +
64 +```
65 +// forget/crates/forget_foo/Cargo.toml
66 +[dependencies]
67 +...
68 +new_dep = { workspace = true }
69 +...
70 +```
71 +
72 +### Adding new crates
73 +
74 +Rust's compilation strategy is largely based on parallelizing at the granularity of crates, so builds can be faster when projects
75 +have more but smaller crates. Where possible it helps to structure crates to minimize dependencies. For example, our various compiler
76 +passes depend on each other in the sense that they often must run in a certain order. However, they often don't need to call each other,
77 +so they can generally be split into crates of similar types of passes, so that those crates can compile in parallel.
78 +
79 +As a rule of thumb, add crates at roughly the granularity of our existing top-level folds. If you have some one-off utility code that
80 +doesn't fit neatly in a crate, add it to `forget_utils` rather than add a one-off crate for it.
81 +
82 +## Running Tests
83 +
84 +Run all tests with the following from the root directory:
85 +
86 +```
87 +cargo test
88 +```
89 +
90 +The majority of our tests will (should) live in the `forget_fixtures` crate, which is a test-only crate that runs compilation end-to-end with snapshot
91 +tests. To run just these tests use:
92 +
93 +```
94 +# quiet version
95 +cargo test -p forget_fixtures
96 +
97 +# without suppressing stdout/stderr output
98 +cargo test -p forget_fixtures -- --nocapture
99 +```
100 +
101 +Another hint is that VSCode will show a "Run test" option if you hover over a test in the source code, this lets you run a single test easily.
102 +The command line will also give you the CLI command to run just that one test.
103 +
104 +## Updating Snapshots
105 +
106 +The above tests make frequent use of snapshot tests. If snapshots do not match the tests will fail with a diff, if the new output is correct you
107 +can accept the changes with:
108 +
109 +```
110 +cargo insta accept
111 +```
112 +
113 +If this command fails, see the note in "first-time setup" about installing `cargo insta`.
\ No newline at end of file
compiler/forget/crates/forget_build_hir/README.md
+1 -1
@@ -1,3 +1,3 @@
1 # Build-HIR
2
3 -This crate converts from ESTree into HIR format as the first phase of compilation.
\ No newline at end of file
3 +This crate converts from `forget_estree` into Forget's HIR format as the first phase of compilation.
\ No newline at end of file
compiler/forget/crates/forget_estree/README.md new
+17
@@ -0,0 +1,17 @@
1 +# forget_estree
2 +
3 +This crate is a Rust representation of the [ESTree format](https://github.com/estree/estree/tree/master) and
4 +popular extenions including JSX and (eventually) Flow and TypeScript.
5 +
6 +This crate is intended as the main interchange format with outside code. A typical integration with Forget
7 +will look as follows:
8 +
9 +1. Host Compiler parses into the host AST format.
10 +2. Host Compiler converts into `forget_estree`.
11 +3. Host Compiler invokes Forget to compile the input, which (conceptually)
12 + returns the resulting code in `forget_estree` format.
13 +4. Host Compiler convert back from `forget_estree` to its host AST format.
14 +
15 +Because Forget is intended to support JavaScript-based toolchains, `forget_estree` is designed to support
16 +accurate serialization to/from estree-compatible JSON. We may also support the Babel AST format
17 +(a variant of ESTree) as well, depending on demand.
\ No newline at end of file
compiler/forget/crates/forget_estree_codegen/README.md new
+4
@@ -0,0 +1,4 @@
1 +# forget_estree_codegen
2 +
3 +This crate is a build dependency for `forget_estree`, and contains codegen logic to produce Rust code to describe the ESTree format
4 +given a JSON schema.
\ No newline at end of file
compiler/forget/crates/forget_estree_swc/README.md new
+4
@@ -0,0 +1,4 @@
1 +# forget_estree_swc
2 +
3 +This crate converts from SWC's AST format into Forget's `forget_estree` format, which is used as the primary
4 +input/type in Forget's public APIs.
\ No newline at end of file
compiler/forget/crates/forget_hir/README.md
+30 -1
@@ -1,3 +1,32 @@
1 # HIR
2
3 -This crate defines the core data structures that Forget uses to represent and compile input programs.
3 +This crate defines the High-level Intermediate Representation (HIR) used by Forget.
4 +
5 +While the name is inspired by Rust Compiler's HIR, Forget's HIR is actually quite different.
6 +Rust's HIR is effectively a compact AST, effectively a slightly more canonical form than the
7 +concrete syntax tree produced by the parser.
8 +
9 +Forget has two goals that are in tension:
10 +
11 +1. Forget needs a detailed understanding of the control-flow semantics and performs sophisticated
12 + data-flow and semantic analysis, all of which benefit from more traditional control-flow graph
13 + representation with flat instruction sequences.
14 +2. At the same time, Forget needs to output code in the original language, and ideally should
15 + produce code that is as similar as possible (for comprehension) and compact (to avoid increasing
16 + bandwidth costs and time to download).
17 +
18 +To satisfy both goals, Forget's HIR uses a hybrid of a traditional intermediate representation and
19 +an AST:
20 +
21 +1. The HIR is a control-flow graph, with one or more basic blocks each of which contains zero or more
22 + instructions and a terminal node. The blocks are stored in reverse postorder so that compiler passes
23 + can iterate the graph and always visit all predecessor blocks before successors, except in the
24 + presence of loops. This allows many passes to complete in a single pass and eases data flow analysis.
25 +2. Unlike a typical intermediate representation, the HIR uses a rich set of high-level terminal nodes.
26 + Rather than just a list of successors, for example, the terminal type is an enum of variants such as
27 + "if", "for", "for-of", "do-while", and other types to represent expression-level control flow in
28 + JavaScript, such as "ternary", "logical", "optional", "sequence", etc. Notably, these terminals contain
29 + named fields with links to successors (eg "for" has fields for the init, test, update, and body blocks)
30 + but also for the "fallthrough" block, ie the block to the code that comes "after" all the logic of
31 + the terminal. This fallthrough allows Forget to retain the shape of the AST and recover it later in
32 + compilation.
\ No newline at end of file
compiler/forget/crates/forget_optimization/README.md new
+3
@@ -0,0 +1,3 @@
1 +# forget_optimization
2 +
3 +Compiler passes that apply various optimizations to improve the performance and/or size of the program.
\ No newline at end of file
compiler/forget/crates/forget_utils/README.md new
+6
@@ -0,0 +1,6 @@
1 +# forget_utils
2 +
3 +This is a catch-all crate for utilities and helper code that doesn't have an obvious home elsewhere.
4 +It is expected that this crate will be depended on by lots of other crates in the project. However,
5 +this crate should generally *not* depend on other workspace crates — that's an indication that the
6 +utility you're adding belongs with the crate that uses the utility.
\ No newline at end of file