external/ton
1
0
Fork 0
mirror of https://github.com/ton-blockchain/ton synced 2025-02-12 19:22:37 +00:00
Code Issues Releases Wiki Activity
ton/tolk/tolk.cpp

88 lines
3.2 KiB
C++
Raw Normal View History

[Tolk] Initial commit of TOLK Language: fork all sources from FunC The Tolk Language will be positioned as "next-generation FunC". It's literally a fork of a FunC compiler, introducing familiar syntax similar to TypeScript, but leaving all low-level optimizations untouched. Note, that FunC sources are partially stored in the parser/ folder (shared with TL/B). In Tolk, nothing is shared. Everything from parser/ is copied into tolk/ folder.
2024-10-31 06:51:07 +00:00
/*
This file is part of TON Blockchain source code.
TON Blockchain is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License
as published by the Free Software Foundation; either version 2
of the License, or (at your option) any later version.
TON Blockchain is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with TON Blockchain. If not, see <http://www.gnu.org/licenses/>.
In addition, as a special exception, the copyright holders give permission
to link the code of portions of this program with the OpenSSL library.
You must obey the GNU General Public License in all respects for all
of the code used other than OpenSSL. If you modify file(s) with this
exception, you may extend this exception to your version of the file(s),
but you are not obligated to do so. If you do not wish to do so, delete this
exception statement from your version. If you delete this exception statement
from all source files in the program, then also delete it here.
*/
#include "tolk.h"
[Tolk] Compilation pipeline, register global symbols in advance Since I've implemented AST, now I can drop forward declarations. Instead, I traverse AST of all files and register global symbols (functions, constants, global vars) as a separate step, in advance. That's why, while converting AST to Expr/Op, all available symbols are already registered. This greatly simplifies "intermediate state" of yet unknown functions and checking them afterward. Redeclaration of local variables (inside the same scope) is now also prohibited.
2024-10-31 07:04:58 +00:00
#include "pipeline.h"
[Tolk] Embedded stdlib.tolk, CompilerState, strict includes Several related changes: - stdlib.tolk is embedded into a distribution (deb package or tolk-js), the user won't have to download it and store as a project file; it's an important step to maintain correct language versioning - stdlib.tolk is auto-included, that's why all its functions are available out of the box - strict includes: you can't use symbol `f` from another file unless you've #include'd this file - drop all C++ global variables holding compilation state, merge them into a single struct CompilerState located at compiler-state.h; for instance, stdlib filename is also there
2024-10-31 07:02:01 +00:00
#include "compiler-state.h"
[Tolk] Initial commit of TOLK Language: fork all sources from FunC The Tolk Language will be positioned as "next-generation FunC". It's literally a fork of a FunC compiler, introducing familiar syntax similar to TypeScript, but leaving all low-level optimizations untouched. Note, that FunC sources are partially stored in the parser/ folder (shared with TL/B). In Tolk, nothing is shared. Everything from parser/ is copied into tolk/ folder.
2024-10-31 06:51:07 +00:00
#include "lexer.h"
[Tolk] Compilation pipeline, register global symbols in advance Since I've implemented AST, now I can drop forward declarations. Instead, I traverse AST of all files and register global symbols (functions, constants, global vars) as a separate step, in advance. That's why, while converting AST to Expr/Op, all available symbols are already registered. This greatly simplifies "intermediate state" of yet unknown functions and checking them afterward. Redeclaration of local variables (inside the same scope) is now also prohibited.
2024-10-31 07:04:58 +00:00
#include "ast.h"
[Tolk] Rewrite the type system from Hindley-Milner to static typing FunC's (and Tolk's before this PR) type system is based on Hindley-Milner. This is a common approach for functional languages, where types are inferred from usage through unification. As a result, type declarations are not necessary: () f(a,b) { return a+b; } // a and b now int, since `+` (int, int) While this approach works for now, problems arise with the introduction of new types like bool, where `!x` must handle both int and bool. It will also become incompatible with int32 and other strict integers. This will clash with structure methods, struggle with proper generics, and become entirely impractical for union types. This PR completely rewrites the type system targeting the future. 1) type of any expression is inferred and never changed 2) this is available because dependent expressions already inferred 3) forall completely removed, generic functions introduced (they work like template functions actually, instantiated while inferring) 4) instantiation `<...>` syntax, example: `t.tupleAt<int>(0)` 5) `as` keyword, for example `t.tupleAt(0) as int` 6) methods binding is done along with type inferring, not before ("before", as worked previously, was always a wrong approach)
2024-12-30 15:31:27 +00:00
#include "type-system.h"
[Tolk] Initial commit of TOLK Language: fork all sources from FunC The Tolk Language will be positioned as "next-generation FunC". It's literally a fork of a FunC compiler, introducing familiar syntax similar to TypeScript, but leaving all low-level optimizations untouched. Note, that FunC sources are partially stored in the parser/ folder (shared with TL/B). In Tolk, nothing is shared. Everything from parser/ is copied into tolk/ folder.
2024-10-31 06:51:07 +00:00
namespace tolk {
[Tolk] Tolk v0.5.0 as FunC v0.5.0 could have been like All changes from PR "FunC v0.5.0": https://github.com/ton-blockchain/ton/pull/1026 Instead of developing FunC, we decided to fork it. BTW, the first Tolk release will be v0.6, a metaphor of FunC v0.5 that missed a chance to occur.
2024-10-31 06:54:05 +00:00
[Tolk] v0.6 syntax: `fun`, `import`, `var`, types on the right, etc. Lots of changes, actually. Most noticeable are: - traditional //comments - #include -> import - a rule "import what you use" - ~ found -> !found (for -1/0) - null() -> null - is_null?(v) -> v == null - throw is a keyword - catch with swapped arguments - throw_if, throw_unless -> assert - do until -> do while - elseif -> else if - drop ifnot, elseifnot - drop rarely used operators A testing framework also appears here. All tests existed earlier, but due to significant syntax changes, their history is useless.
2024-10-31 07:11:41 +00:00
void on_assertion_failed(const char *description, const char *file_name, int line_number) {
std::string message = static_cast<std::string>("Assertion failed at ") + file_name + ":" + std::to_string(line_number) + ": " + description;
#ifdef TOLK_DEBUG
#ifdef __arm64__
// when developing, it's handy when the debugger stops on assertion failure (stacktraces and watches are available)
std::cerr << message << std::endl;
__builtin_debugtrap();
#endif
#endif
throw Fatal(std::move(message));
}
[Tolk] Compilation pipeline, register global symbols in advance Since I've implemented AST, now I can drop forward declarations. Instead, I traverse AST of all files and register global symbols (functions, constants, global vars) as a separate step, in advance. That's why, while converting AST to Expr/Op, all available symbols are already registered. This greatly simplifies "intermediate state" of yet unknown functions and checking them afterward. Redeclaration of local variables (inside the same scope) is now also prohibited.
2024-10-31 07:04:58 +00:00
int tolk_proceed(const std::string &entrypoint_filename) {
[Tolk] Rewrite the type system from Hindley-Milner to static typing FunC's (and Tolk's before this PR) type system is based on Hindley-Milner. This is a common approach for functional languages, where types are inferred from usage through unification. As a result, type declarations are not necessary: () f(a,b) { return a+b; } // a and b now int, since `+` (int, int) While this approach works for now, problems arise with the introduction of new types like bool, where `!x` must handle both int and bool. It will also become incompatible with int32 and other strict integers. This will clash with structure methods, struggle with proper generics, and become entirely impractical for union types. This PR completely rewrites the type system targeting the future. 1) type of any expression is inferred and never changed 2) this is available because dependent expressions already inferred 3) forall completely removed, generic functions introduced (they work like template functions actually, instantiated while inferring) 4) instantiation `<...>` syntax, example: `t.tupleAt<int>(0)` 5) `as` keyword, for example `t.tupleAt(0) as int` 6) methods binding is done along with type inferring, not before ("before", as worked previously, was always a wrong approach)
2024-12-30 15:31:27 +00:00
type_system_init();
[Tolk] Initial commit of TOLK Language: fork all sources from FunC The Tolk Language will be positioned as "next-generation FunC". It's literally a fork of a FunC compiler, introducing familiar syntax similar to TypeScript, but leaving all low-level optimizations untouched. Note, that FunC sources are partially stored in the parser/ folder (shared with TL/B). In Tolk, nothing is shared. Everything from parser/ is copied into tolk/ folder.
2024-10-31 06:51:07 +00:00
define_builtins();
[Tolk] Rewrite lexer, spaces are not mandatory anymore A new lexer is noticeably faster and memory efficient (although splitting a file to tokens is negligible in a whole pipeline). But the purpose of rewriting lexer was not just to speed up, but to allow writing code without spaces: `2+2` is now 4, not a valid identifier as earlier. The variety of symbols allowed in identifier has greatly reduced and is now similar to other languages. SrcLocation became 8 bytes on stack everywhere. Command-line flags were also reworked: - the input for Tolk compiler is only a single file now, it's parsed, and parsing continues while new #include are resolved - flags like -A -P and so on are no more needed, actually
2024-10-31 06:59:23 +00:00
lexer_init();
[Tolk] Initial commit of TOLK Language: fork all sources from FunC The Tolk Language will be positioned as "next-generation FunC". It's literally a fork of a FunC compiler, introducing familiar syntax similar to TypeScript, but leaving all low-level optimizations untouched. Note, that FunC sources are partially stored in the parser/ folder (shared with TL/B). In Tolk, nothing is shared. Everything from parser/ is copied into tolk/ folder.
2024-10-31 06:51:07 +00:00
[Tolk] Completely rework stdlib: multiple files and renaming - split stdlib.tolk into multiple files (tolk-stdlib/ folder) (the "core" common.tolk is auto-imported, the rest are needed to be explicitly imported like "@stdlib/tvm-dicts.tolk") - all functions were renamed to long and clear names - new naming is camelCase
2024-10-31 07:16:19 +00:00
// on any error, an exception is thrown, and the message is printed out below
// (currently, only a single error can be printed)
[Tolk] Initial commit of TOLK Language: fork all sources from FunC The Tolk Language will be positioned as "next-generation FunC". It's literally a fork of a FunC compiler, introducing familiar syntax similar to TypeScript, but leaving all low-level optimizations untouched. Note, that FunC sources are partially stored in the parser/ folder (shared with TL/B). In Tolk, nothing is shared. Everything from parser/ is copied into tolk/ folder.
2024-10-31 06:51:07 +00:00
try {
[Tolk] Rewrite the type system from Hindley-Milner to static typing FunC's (and Tolk's before this PR) type system is based on Hindley-Milner. This is a common approach for functional languages, where types are inferred from usage through unification. As a result, type declarations are not necessary: () f(a,b) { return a+b; } // a and b now int, since `+` (int, int) While this approach works for now, problems arise with the introduction of new types like bool, where `!x` must handle both int and bool. It will also become incompatible with int32 and other strict integers. This will clash with structure methods, struggle with proper generics, and become entirely impractical for union types. This PR completely rewrites the type system targeting the future. 1) type of any expression is inferred and never changed 2) this is available because dependent expressions already inferred 3) forall completely removed, generic functions introduced (they work like template functions actually, instantiated while inferring) 4) instantiation `<...>` syntax, example: `t.tupleAt<int>(0)` 5) `as` keyword, for example `t.tupleAt(0) as int` 6) methods binding is done along with type inferring, not before ("before", as worked previously, was always a wrong approach)
2024-12-30 15:31:27 +00:00
pipeline_discover_and_parse_sources("@stdlib/common.tolk", entrypoint_filename);
[Tolk] Compilation pipeline, register global symbols in advance Since I've implemented AST, now I can drop forward declarations. Instead, I traverse AST of all files and register global symbols (functions, constants, global vars) as a separate step, in advance. That's why, while converting AST to Expr/Op, all available symbols are already registered. This greatly simplifies "intermediate state" of yet unknown functions and checking them afterward. Redeclaration of local variables (inside the same scope) is now also prohibited.
2024-10-31 07:04:58 +00:00
[Tolk] Rewrite the type system from Hindley-Milner to static typing FunC's (and Tolk's before this PR) type system is based on Hindley-Milner. This is a common approach for functional languages, where types are inferred from usage through unification. As a result, type declarations are not necessary: () f(a,b) { return a+b; } // a and b now int, since `+` (int, int) While this approach works for now, problems arise with the introduction of new types like bool, where `!x` must handle both int and bool. It will also become incompatible with int32 and other strict integers. This will clash with structure methods, struggle with proper generics, and become entirely impractical for union types. This PR completely rewrites the type system targeting the future. 1) type of any expression is inferred and never changed 2) this is available because dependent expressions already inferred 3) forall completely removed, generic functions introduced (they work like template functions actually, instantiated while inferring) 4) instantiation `<...>` syntax, example: `t.tupleAt<int>(0)` 5) `as` keyword, for example `t.tupleAt(0) as int` 6) methods binding is done along with type inferring, not before ("before", as worked previously, was always a wrong approach)
2024-12-30 15:31:27 +00:00
pipeline_register_global_symbols();
pipeline_resolve_identifiers_and_assign_symbols();
pipeline_calculate_rvalue_lvalue();
pipeline_detect_unreachable_statements();
pipeline_infer_types_and_calls_and_fields();
pipeline_refine_lvalue_for_mutate_arguments();
pipeline_check_rvalue_lvalue();
pipeline_check_pure_impure_operations();
pipeline_constant_folding();
[Tolk] `bool` type (-1/0 int under the hood) Comparison operators `== / >= /...` return `bool`. Logical operators `&& ||` return bool. Constants `true` and `false` have the `bool` type. Lots of stdlib functions return `bool`, not `int`. Operator `!x` supports both `int` and `bool`. Condition of `if` accepts both `int` and `bool`. Arithmetic operators are restricted to integers. Logical `&&` and `||` accept both `bool` and `int`. No arithmetic operations with bools allowed (only bitwise and logical).
2025-01-13 08:21:24 +00:00
pipeline_optimize_boolean_expressions();
[Tolk] Rewrite the type system from Hindley-Milner to static typing FunC's (and Tolk's before this PR) type system is based on Hindley-Milner. This is a common approach for functional languages, where types are inferred from usage through unification. As a result, type declarations are not necessary: () f(a,b) { return a+b; } // a and b now int, since `+` (int, int) While this approach works for now, problems arise with the introduction of new types like bool, where `!x` must handle both int and bool. It will also become incompatible with int32 and other strict integers. This will clash with structure methods, struggle with proper generics, and become entirely impractical for union types. This PR completely rewrites the type system targeting the future. 1) type of any expression is inferred and never changed 2) this is available because dependent expressions already inferred 3) forall completely removed, generic functions introduced (they work like template functions actually, instantiated while inferring) 4) instantiation `<...>` syntax, example: `t.tupleAt<int>(0)` 5) `as` keyword, for example `t.tupleAt(0) as int` 6) methods binding is done along with type inferring, not before ("before", as worked previously, was always a wrong approach)
2024-12-30 15:31:27 +00:00
pipeline_convert_ast_to_legacy_Expr_Op();
[Tolk] Compilation pipeline, register global symbols in advance Since I've implemented AST, now I can drop forward declarations. Instead, I traverse AST of all files and register global symbols (functions, constants, global vars) as a separate step, in advance. That's why, while converting AST to Expr/Op, all available symbols are already registered. This greatly simplifies "intermediate state" of yet unknown functions and checking them afterward. Redeclaration of local variables (inside the same scope) is now also prohibited.
2024-10-31 07:04:58 +00:00
pipeline_find_unused_symbols();
[Tolk] Rewrite the type system from Hindley-Milner to static typing FunC's (and Tolk's before this PR) type system is based on Hindley-Milner. This is a common approach for functional languages, where types are inferred from usage through unification. As a result, type declarations are not necessary: () f(a,b) { return a+b; } // a and b now int, since `+` (int, int) While this approach works for now, problems arise with the introduction of new types like bool, where `!x` must handle both int and bool. It will also become incompatible with int32 and other strict integers. This will clash with structure methods, struggle with proper generics, and become entirely impractical for union types. This PR completely rewrites the type system targeting the future. 1) type of any expression is inferred and never changed 2) this is available because dependent expressions already inferred 3) forall completely removed, generic functions introduced (they work like template functions actually, instantiated while inferring) 4) instantiation `<...>` syntax, example: `t.tupleAt<int>(0)` 5) `as` keyword, for example `t.tupleAt(0) as int` 6) methods binding is done along with type inferring, not before ("before", as worked previously, was always a wrong approach)
2024-12-30 15:31:27 +00:00
pipeline_generate_fif_output_to_std_cout();
[Tolk] Rewrite lexer, spaces are not mandatory anymore A new lexer is noticeably faster and memory efficient (although splitting a file to tokens is negligible in a whole pipeline). But the purpose of rewriting lexer was not just to speed up, but to allow writing code without spaces: `2+2` is now 4, not a valid identifier as earlier. The variety of symbols allowed in identifier has greatly reduced and is now similar to other languages. SrcLocation became 8 bytes on stack everywhere. Command-line flags were also reworked: - the input for Tolk compiler is only a single file now, it's parsed, and parsing continues while new #include are resolved - flags like -A -P and so on are no more needed, actually
2024-10-31 06:59:23 +00:00
[Tolk] Embedded stdlib.tolk, CompilerState, strict includes Several related changes: - stdlib.tolk is embedded into a distribution (deb package or tolk-js), the user won't have to download it and store as a project file; it's an important step to maintain correct language versioning - stdlib.tolk is auto-included, that's why all its functions are available out of the box - strict includes: you can't use symbol `f` from another file unless you've #include'd this file - drop all C++ global variables holding compilation state, merge them into a single struct CompilerState located at compiler-state.h; for instance, stdlib filename is also there
2024-10-31 07:02:01 +00:00
return 0;
[Tolk] Initial commit of TOLK Language: fork all sources from FunC The Tolk Language will be positioned as "next-generation FunC". It's literally a fork of a FunC compiler, introducing familiar syntax similar to TypeScript, but leaving all low-level optimizations untouched. Note, that FunC sources are partially stored in the parser/ folder (shared with TL/B). In Tolk, nothing is shared. Everything from parser/ is copied into tolk/ folder.
2024-10-31 06:51:07 +00:00
} catch (Fatal& fatal) {
[Tolk] Embedded stdlib.tolk, CompilerState, strict includes Several related changes: - stdlib.tolk is embedded into a distribution (deb package or tolk-js), the user won't have to download it and store as a project file; it's an important step to maintain correct language versioning - stdlib.tolk is auto-included, that's why all its functions are available out of the box - strict includes: you can't use symbol `f` from another file unless you've #include'd this file - drop all C++ global variables holding compilation state, merge them into a single struct CompilerState located at compiler-state.h; for instance, stdlib filename is also there
2024-10-31 07:02:01 +00:00
std::cerr << "fatal: " << fatal << std::endl;
[Tolk] Initial commit of TOLK Language: fork all sources from FunC The Tolk Language will be positioned as "next-generation FunC". It's literally a fork of a FunC compiler, introducing familiar syntax similar to TypeScript, but leaving all low-level optimizations untouched. Note, that FunC sources are partially stored in the parser/ folder (shared with TL/B). In Tolk, nothing is shared. Everything from parser/ is copied into tolk/ folder.
2024-10-31 06:51:07 +00:00
return 2;
[Tolk] Rewrite lexer, spaces are not mandatory anymore A new lexer is noticeably faster and memory efficient (although splitting a file to tokens is negligible in a whole pipeline). But the purpose of rewriting lexer was not just to speed up, but to allow writing code without spaces: `2+2` is now 4, not a valid identifier as earlier. The variety of symbols allowed in identifier has greatly reduced and is now similar to other languages. SrcLocation became 8 bytes on stack everywhere. Command-line flags were also reworked: - the input for Tolk compiler is only a single file now, it's parsed, and parsing continues while new #include are resolved - flags like -A -P and so on are no more needed, actually
2024-10-31 06:59:23 +00:00
} catch (ParseError& error) {
[Tolk] Embedded stdlib.tolk, CompilerState, strict includes Several related changes: - stdlib.tolk is embedded into a distribution (deb package or tolk-js), the user won't have to download it and store as a project file; it's an important step to maintain correct language versioning - stdlib.tolk is auto-included, that's why all its functions are available out of the box - strict includes: you can't use symbol `f` from another file unless you've #include'd this file - drop all C++ global variables holding compilation state, merge them into a single struct CompilerState located at compiler-state.h; for instance, stdlib filename is also there
2024-10-31 07:02:01 +00:00
std::cerr << error << std::endl;
[Tolk] Initial commit of TOLK Language: fork all sources from FunC The Tolk Language will be positioned as "next-generation FunC". It's literally a fork of a FunC compiler, introducing familiar syntax similar to TypeScript, but leaving all low-level optimizations untouched. Note, that FunC sources are partially stored in the parser/ folder (shared with TL/B). In Tolk, nothing is shared. Everything from parser/ is copied into tolk/ folder.
2024-10-31 06:51:07 +00:00
return 2;
[Tolk] Implement AST: intermediate representation of tolk files Now, the whole .tolk file can be loaded as AST tree and then converted to Expr/Op. This gives a great ability to implement AST transformations. In the future, more and more code analysis will be moved out of legacy to AST-level.
2024-10-31 07:03:33 +00:00
} catch (UnexpectedASTNodeType& error) {
std::cerr << "fatal: " << error.what() << std::endl;
std::cerr << "It's a compiler bug, please report to developers" << std::endl;
return 2;
[Tolk] Initial commit of TOLK Language: fork all sources from FunC The Tolk Language will be positioned as "next-generation FunC". It's literally a fork of a FunC compiler, introducing familiar syntax similar to TypeScript, but leaving all low-level optimizations untouched. Note, that FunC sources are partially stored in the parser/ folder (shared with TL/B). In Tolk, nothing is shared. Everything from parser/ is copied into tolk/ folder.
2024-10-31 06:51:07 +00:00
}
}
} // namespace tolk
Reference in a new issue Copy permalink