AI 时代为什么还是需要理解语法树？

Tree Sitter NG

Next generation Tree Sitter Java binding.

Getting started

Add dependencies to your build.gradle or pom.xml.

// Gradle
dependencies {
    // add tree sitter
    implementation 'io.github.bonede:tree-sitter:$VERSION'
    // add json parser
    implementation 'io.github.bonede:tree-sitter-json:$VERSION'
}

<!-- Maven -->
<dpendencies>
    <!-- add tree sitter -->
    <dependency>
        <groupId>io.github.bonede</groupId>
        <artifactId>tree-sitter</artifactId>
        <version>$VERSION</version>
    </dependency>
    <!-- add json parser -->
    <dependency>
        <groupId>io.github.bonede</groupId>
        <artifactId>tree-sitter-json</artifactId>
        <version>$VERSION</version>
    </dependency>
</dpendencies>

版本：

Name	Version
tree-sitter-agda	v1.3.3
tree-sitter-bash	v0.25.1
tree-sitter-c	    v0.24.1
tree-sitter-c-sharp	v0.23.1
tree-sitter-cpp	   v0.23.4
tree-sitter-css	   v0.25.0
tree-sitter-embedded-template	v0.25.0
tree-sitter-go	v0.25.0
tree-sitter-haskell	v0.23.1
tree-sitter-html	v0.23.2
tree-sitter-java	v0.23.5
tree-sitter-javascript	v0.25.0
tree-sitter-json	v0.24.8
tree-sitter-julia	v0.25.0
tree-sitter-ocaml	v0.23.2
tree-sitter-php	  v0.24.2
tree-sitter-python	v0.25.0
tree-sitter-regex	  v1.0.0 
tree-sitter-ruby	v0.23.1
tree-sitter-rust	v0.24.0
tree-sitter-scala	v0.24.0
tree-sitter-typescript	v0.23.2
tree-sitter-verilog	    v1.0.3
tree-sitter-tsx	不存在！

以 tree-sitter-java 为例子，maven 引入方式如下，把 v 去掉

<dependency>
    <groupId>io.github.bonede</groupId>
    <artifactId>tree-sitter-java</artifactId>
    <version>0.23.4</version>
</dependency>

Start hacking!

// imports are omitted
class Main {
    public static void main(String[] args) {
        TSParser parser = new TSParser();
        // Use `TSLanguage.load` instead if you would like to load parsers as shared object(.so, .dylib, or .dll).
        // TSLanguage.load("path/to/languane/shared/object", "tree_sitter_some_lang");
        TSLanguage json = new TreeSitterJson();
        parser.setLanguage(json);
        TSTree tree = parser.parseString(null, "[1, null]");
        TSNode rootNode = tree.getRootNode();
        TSNode arrayNode = rootNode.getNamedChild(0);
        TSNode numberNode = arrayNode.getNamedChild(0);
    }
}

Features

• 100% Tree Sitter API coverage.
• Easy to bootstrap cross compiling environments powered by Zig.
• Built-in official parsers.
• Load parsers as shared object from disk.

Supported CPUs and OSes

• x86_64-windows
• x86_64-macos
• aarch64-macos
• x86_64-linux
• aarch64-linux

Built-in official parsers

| Name | Version | |———————————|———————————————————————————————————| | tree-sitter-agda | | | tree-sitter-bash | | | tree-sitter-c | | | tree-sitter-c-sharp | | | tree-sitter-cpp | | | tree-sitter-css | | | tree-sitter-embedded-template | | | tree-sitter-go | | | tree-sitter-haskell | | | tree-sitter-html | | | tree-sitter-java | | | tree-sitter-javascript | | | tree-sitter-json | | | tree-sitter-julia | | | tree-sitter-ocaml | | | tree-sitter-php | | | tree-sitter-python | | | tree-sitter-regex | | | tree-sitter-ruby | | | tree-sitter-rust | | | tree-sitter-scala | | | tree-sitter-tsx | | | tree-sitter-typescript | | | tree-sitter-verilog | |

API Tour

class Main {
    public static void main(String[] args) {
        String jsonSource = "[1, null]";
        TSParser parser = new TSParser();
        TSLanguage json = new TreeSitterJson();
        // set language parser
        parser.setLanguage(json);
        // parser with string input
        parser.parseString(null, jsonSource);
        parser.reset();
        // or parser with encoding
        parser.parseStringEncoding(null, JSON_SRC, TSInputEncoding.TSInputEncodingUTF8);
        parser.reset();
        // or parser with custom reader
        byte[] buffer = new byte[1024];
        TSReader reader = (buf, offset, position) -> {
            if(offset >= jsonSource.length()){
                return 0;
            }
            ByteBuffer charBuffer = ByteBuffer.wrap(buf);
            charBuffer.put(jsonSource.getBytes());
            return jsonSource.length();
        };
        TSTree tree = parser.parse(buffer, null, reader, TSInputEncoding.TSInputEncodingUTF8);
        // traverse the AST tree with DOM like APIs
        TSNode rootNode = tree.getRootNode();
        TSNode arrayNode = rootNode.getNamedChild(0);
        // or travers the AST with cursor
        TSTreeCursor rootCursor = new TSTreeCursor(rootNode);
        rootCursor.gotoFirstChild();
        // or query the AST with S-expression
        TreeSitterQuery query = new TSQuery(json, "((document) @root)");
        TSQueryCursor cursor = new TSQueryCursor();
        cursor.exec(query, rootNode);
        SQueryMatch match = new TSQueryMatch();
        while(cursor.nextMatch(match)){
            // do something with the match
        }
        // debug the parser with a logger
        TSLogger logger = (type, message) -> {
            System.out.println(message);
        };
        parser.setLogger(logger);
        // or output the AST tree as DOT graph
        File dotFile = File.createTempFile("json", ".dot");
        parser.printDotGraphs(dotFile);
    }
}

参考资料

https://github.com/bonede/tree-sitter-ng

• Tree Sitter NG
• Getting started
• Features
• Supported CPUs and OSes
• Built-in official parsers
• API Tour
• 参考资料