* Motivation: We want to make it easy to generate variants of tests. Often, we would like to * have a set of tests, corresponding to a set of types, a set of operators, a set of constants, etc. Writing all * the tests by hand is cumbersome or even impossible. When generating such tests with scripts, it would be * preferable if the code generation happens automatically, and the generator script was checked into the code * base. Code generation can go beyond simple regression tests, and one might want to generate random code from * a list of possible templates, to fuzz individual Java features and compiler optimizations. * *
* The Template Framework provides a facility to generate code with Templates. A Template is essentially a list * of tokens that are concatenated (i.e. rendered) to a {@link String}. The Templates can have "holes", which are * filled (replaced) by different values at each Template instantiation. For example, these "holes" can * be filled with different types, operators or constants. Templates can also be nested, allowing a modular * use of Templates. * *
* Once we rendered the source code to a {@link String}, we can compile it with the {@link CompileFramework}. * *
* Example: * The following snippets are from the example test {@code TestAdvanced.java}. * First, we define a template that generates a {@code @Test} method for a given type, operator and * constant generator. We define two constants {@code con1} and {@code con2}, and then use a multiline * string with hashtags {@code #} (i.e. "holes") that are then replaced by the template arguments and the * {@link #let} definitions. * *
* {@snippet lang=java : * var testTemplate = Template.make("typeName", "operator", "generator", (String typeName, String operator, MyGenerator generator) -> scope( * let("con1", generator.next()), * let("con2", generator.next()), * """ * // #typeName #operator #con1 #con2 * public static #typeName $GOLD = $test(); * * @Test * public static #typeName $test() { * return (#typeName)(#con1 #operator #con2); * } * * @Check(test = "$test") * public static void $check(#typeName result) { * Verify.checkEQ(result, $GOLD); * } * """ * )); * } * *
* To get an executable test, we define a {@link Template} that produces a class scope with a main method. The Template * takes a list of types, and calls the {@code testTemplate} defined above for each type and operator. We use * the {@link TestFramework} to call our {@code @Test} methods. * *
* {@snippet lang=java :
* var classTemplate = Template.make("types", (List
* Finally, we generate the list of types, and pass it to the class template:
*
*
* {@snippet lang=java :
* List
* Details:
*
* A {@link Template} can have zero or more arguments. A template can be created with {@code make} methods like
* {@link Template#make(String, Function)}. For each number of arguments there is an implementation
* (e.g. {@link Template.TwoArgs} for two arguments). This allows the use of generics for the
* {@link Template} argument types which enables type checking of the {@link Template} arguments.
* It is currently only allowed to use up to three arguments.
*
*
* A {@link Template} can be rendered to a {@link String} (e.g. {@link Template.ZeroArgs#render()}).
* Alternatively, we can generate a {@link Token} (more specifically, a {@link TemplateToken}) with {@code asToken()}
* (e.g. {@link Template.ZeroArgs#asToken()}), and use the {@link Token} inside another {@link Template#scope}.
*
*
* Ideally, we would have used string templates to inject these Template
* arguments into the strings. But since string templates are not (yet) available, the Templates provide
* hashtag replacements in the {@link String}s: the Template argument names are captured, and
* the argument values automatically replace any {@code "#name"} in the {@link String}s. See the different overloads
* of {@link #make} for examples. Additional hashtag replacements can be defined with {@link #let}. If a "#" is needed
* in the code, hashtag replacmement can be escaped by writing two hashtags, i.e. "##" will render as "#".
* We have decided to keep hashtag replacements constrained to the scope of one Template. They
* do not escape to outer or inner Template uses. If one needs to pass values to inner Templates,
* this can be done with Template arguments. Keeping hashtag replacements local to Templates
* has the benefit that there is no conflict in recursive templates, where outer and inner Templates
* define the same hashtag replacement.
*
*
* When using nested Templates, there can be collisions with identifiers (e.g. variable names and method names).
* For this, Templates provide dollar replacements, which automatically rename any
* {@code "$name"} in the {@link String} with a {@code "name_ID"}, where the {@code "ID"} is unique for every use of
* a Template. The dollar replacement can also be captured with {@link #$}, and passed to nested
* Templates, which allows sharing of these identifier names between Templates. Similar to hashtag replacements,
* dollars can be escaped by doubling up, i.e. "$$" renders as "$".
*
*
* The dollar and hashtag names must have at least one character. The first character must be a letter
* or underscore (i.e. {@code a-zA-Z_}), the other characters can also be digits (i.e. {@code a-zA-Z0-9_}).
* One can use them with or without curly braces, e.g. {@code #name}, {@code #{name}}, {@code $name}, or
* {@code #{name}}.
*
*
* Code generation can involve keeping track of fields and variables, as well as the scopes in which they
* are available, and if they are mutable or immutable. We model fields and variables with {@link DataName}s,
* which we can add to the current scope with {@link #addDataName}. We can access the {@link DataName}s with
* {@link #dataNames}. We can filter for {@link DataName}s of specific {@link DataName.Type}s, and then
* we can call {@link DataName.FilteredSet#count}, {@link DataName.FilteredSet#sample},
* {@link DataName.FilteredSet#toList}, etc. There are many use-cases for this mechanism, especially
* facilitating communication between the code of outer and inner {@link Template}s. Especially for fuzzing,
* it may be useful to be able to add fields and variables, and sample them randomly, to create a random data
* flow graph.
*
*
* Similarly, we may want to model method and class names, and possibly other structural names. We model
* these names with {@link StructuralName}, which works analogously to {@link DataName}, except that they
* are not concerned about mutability.
*
*
* Code generation can involve keeping track of scopes in the code (e.g. liveness and availability of
* {@link DataName}s) and of the hashtag replacements in the templates. The {@link ScopeToken} serves
* this purpose, and allows the definition of transparent scopes (e.g. {@link #transparentScope}) and
* non-transparent scopes (e.g. {@link #scope}).
*
*
* In some cases, we may be deeper nested in templates and scopes, and would like to reach "back" or
* to outer scopes. This is possible with {@link Hook#anchor}ing in some outer scope, and later
* {@link Hook#insert}ing from an inner scope to the scope of the anchoring. For example, while
* generating code in a method, one can reach out to the scope of the class, and insert a new field,
* or define a utility method.
*
*
* A {@link TemplateBinding} allows the recursive use of Templates. With the indirection of such a binding,
* a Template can reference itself.
*
*
* The writer of recursive {@link Template}s must ensure that this recursion terminates. To unify the
* approach across {@link Template}s, we introduce the concept of {@link #fuel}. Templates are rendered starting
* with a limited amount of {@link #fuel} (default: 100, see {@link #DEFAULT_FUEL}), which is decreased at each
* Template nesting by a certain amount (default: 10, see {@link #DEFAULT_FUEL_COST}). The default fuel for a
* template can be changed when we {@code render()} it (e.g. {@link ZeroArgs#render(float)}) and the default
* fuel cost with {@link #setFuelCost}) when defining the {@link #scope(Object...)}. Recursive templates are
* supposed to terminate once the {@link #fuel} is depleted (i.e. reaches zero).
*
*
* A note from the implementor to the user: We have decided to implement the Template Framework using
* a functional (lambdas) and data-oriented (tokens) model. The consequence is that there are three
* orders in template rendering: (1) the execution order in lambdas, where we usually assemble the
* tokens and pass them to some scope ({@link ScopeToken}) as arguments. (2) the token evaluation
* order, which occurs in the order of how tokens are listed in a scope. By design, the token order
* is the same order as execution in lambdas. To keep the lambda and token order in sync, most of the
* queries about the state of code generation, such as {@link DataName}s and {@link Hook}s cannot
* return the values immediately, but have to be expressed as tokens. If we had a mix of tokens and
* immediate queries, then the immediate queries would "float" by the tokens, because the immediate
* queries are executed during the lambda execution, but the tokens are only executed later. Having
* to express everything as tokens can be a little more cumbersome (e.g. sample requires a lambda
* that captures the {@link DataName}, and sample does not return the {@link DataName} directly).
* But this ensures that reasoning about execution order is relatively straight forward, namely in
* the order of the specified tokens. (3) the final code order is the same as the lambda and token
* order, except when using {@link Hook#insert}, which places the code at the innermost {@link Hook#anchor}.
*
*
* More examples for these functionalities can be found in {@code TestTutorial.java}, {@code TestSimple.java},
* and {@code TestAdvanced.java}, which all produce compilable Java code. Additional examples can be found in
* the tests, such as {@code TestTemplate.java} and {@code TestFormat.java}, which do not necessarily generate
* valid Java code, but generate deterministic Strings which are easier to verify, and may also serve as a
* reference when learning about these functionalities.
*/
public sealed interface Template permits Template.ZeroArgs,
Template.OneArg,
Template.TwoArgs,
Template.ThreeArgs {
/**
* A {@link Template} with no arguments.
*
* @param function The {@link Supplier} that creates the {@link ScopeToken}.
*/
record ZeroArgs(Supplier
* Example:
* {@snippet lang=java :
* var template = Template.make(() -> scope(
* """
* Multi-line string or other tokens.
* """
* ));
* }
*
* @param scope The {@link ScopeToken} created by {@link Template#scope}.
* @return A {@link Template} with zero arguments.
*/
static Template.ZeroArgs make(Supplier
* Here is an example with template argument {@code 'a'}, captured once as string name
* for use in hashtag replacements, and captured once as lambda argument with the corresponding type
* of the generic argument.
* {@snippet lang=java :
* var template = Template.make("a", (Integer a) -> scope(
* """
* Multi-line string or other tokens.
* We can use the hashtag replacement #a to directly insert the String value of a.
* """,
* "We can also use the captured parameter of a: " + a
* ));
* }
*
* @param scope The {@link ScopeToken} created by {@link Template#scope}.
* @param
* Here is an example with template arguments {@code 'a'} and {@code 'b'}, captured once as string names
* for use in hashtag replacements, and captured once as lambda arguments with the corresponding types
* of the generic arguments.
* {@snippet lang=java :
* var template = Template.make("a", "b", (Integer a, String b) -> scope(
* """
* Multi-line string or other tokens.
* We can use the hashtag replacement #a and #b to directly insert the String value of a and b.
* """,
* "We can also use the captured parameter of a and b: " + a + " and " + b
* ));
* }
*
* @param scope The {@link ScopeToken} created by {@link Template#scope}.
* @param
* The scope is formed from a list of tokens, which can be {@link String}s,
* boxed primitive types (for example {@link Integer} or auto-boxed {@code int}),
* any {@link Token}, or {@link List}s of any of these.
*
*
* If you require a scope that is either fully transparent (i.e. everything escapes)
* or only restricts a specific kind to not escape, consider using one of the other
* provided scopes: {@link #transparentScope}, {@link #nameScope}, {@link #hashtagScope},
* or {@link #setFuelCostScope}. A "scope-transparency-matrix" can also be found in
* the interface comment for {@link Template}.
*
*
* The most common use of {@link #scope} is in the construction of templates:
*
*
* {@snippet lang=java :
* var template = Template.make(() -> scope(
* """
* Multi-line string
* """,
* "normal string ", Integer.valueOf(3), 3, Float.valueOf(1.5f), 1.5f,
* List.of("abc", "def"),
* nestedTemplate.asToken(42)
* ));
* }
*
*
* Note that regardless of the chosen scope for {@code Template.make},
* hashtag-replacements and {@link #setFuelCost} are always implicitly
* non-transparent (i.e. non-escaping). For example, {@link #let} will
* not escape the template scope even when using {@link #transparentScope}.
* As a default, it is recommended to use {@link #scope} for
* {@code Template.make} since in most cases template scopes align with
* code scopes that are non-transparent for fields, variables, etc. In
* rare cases, where the scope of the template needs to be transparent
* (e.g. because we need to insert a variable or field into an outer scope),
* it is recommended to use {@link #transparentScope}. This allows to make
* {@link DataName}s and {@link StructuralName}s available outside this
* template crossing the template boundary.
*
*
* We can also use nested scopes inside of templates:
*
*
* {@snippet lang=java :
* var template = Template.make(() -> scope(
* // CODE1: some code in the outer scope
* scope(
* // CODE2: some code in the inner scope. Names, hashtags and setFuelCost
* // do not escape the inner scope.
* ),
* // CODE3: more code in the outer scope, names and hashtags from CODE2 are
* // not available anymore because of the non-transparent "scope".
* transparentScope(
* // CODE4: some code in the inner "transparentScope". Names, hashtags and setFuelCost
* // escape the "transparentScope" and are still available after the "transparentScope"
* // closes.
* )
* // CODE5: we still have access to names and hashtags from CODE4.
* ));
* }
*
* @param tokens A list of tokens, which can be {@link String}s, boxed primitive types
* (for example {@link Integer}), any {@link Token}, or {@link List}s
* of any of these.
* @return The {@link ScopeToken} which captures the list of validated {@link Token}s.
* @throws IllegalArgumentException if the list of tokens contains an unexpected object.
*/
static ScopeToken scope(Object... tokens) {
return new ScopeTokenImpl(TokenParser.parse(tokens), false, false, false);
}
/**
* Creates a {@link ScopeToken} that represents a completely transparent scope.
* This means that {@link DataName}s, {@link StructuralName}s,
* hashtag-replacements and {@link #setFuelCost} declared inside the scope will be available
* in the outer scope.
* The scope is formed from a list of tokens, which can be {@link String}s,
* boxed primitive types (for example {@link Integer} or auto-boxed {@code int}),
* any {@link Token}, or {@link List}s of any of these.
*
*
* If you require a scope that is non-transparent (i.e. nothing escapes) or only restricts
* a specific kind to not escape, consider using one of the other provided scopes:
* {@link #scope}, {@link #nameScope}, {@link #hashtagScope}, or {@link #setFuelCostScope}.
* A "scope-transparency-matrix" can also be found in the interface comment for {@link Template}.
*
* @param tokens A list of tokens, which can be {@link String}s, boxed primitive types
* (for example {@link Integer}), any {@link Token}, or {@link List}s
* of any of these.
* @return The {@link ScopeToken} which captures the list of validated {@link Token}s.
* @throws IllegalArgumentException if the list of tokens contains an unexpected object.
*/
static ScopeToken transparentScope(Object... tokens) {
return new ScopeTokenImpl(TokenParser.parse(tokens), true, true, true);
}
/**
* Creates a {@link ScopeToken} that represents a scope that is non-transparent for
* {@link DataName}s and {@link StructuralName}s (i.e. cannot escape), but
* transparent for hashtag-replacements and {@link #setFuelCost} (i.e. available
* in outer scope).
*
*
* The scope is formed from a list of tokens, which can be {@link String}s,
* boxed primitive types (for example {@link Integer} or auto-boxed {@code int}),
* any {@link Token}, or {@link List}s of any of these.
*
*
* If you require a scope that is transparent or uses a different restriction, consider
* using one of the other provided scopes: {@link #scope}, {@link #transparentScope},
* {@link #hashtagScope}, or {@link #setFuelCostScope}. A "scope-transparency-matrix" can
* also be found in the interface comment for {@link Template}.
*
* @param tokens A list of tokens, which can be {@link String}s, boxed primitive types
* (for example {@link Integer}), any {@link Token}, or {@link List}s
* of any of these.
* @return The {@link ScopeToken} which captures the list of validated {@link Token}s.
* @throws IllegalArgumentException if the list of tokens contains an unexpected object.
*/
static ScopeToken nameScope(Object... tokens) {
return new ScopeTokenImpl(TokenParser.parse(tokens), false, true, true);
}
/**
* Creates a {@link ScopeToken} that represents a scope that is non-transparent for
* hashtag-replacements (i.e. cannot escape), but transparent for {@link DataName}s
* and {@link StructuralName}s and {@link #setFuelCost} (i.e. available in outer scope).
*
*
* The scope is formed from a list of tokens, which can be {@link String}s,
* boxed primitive types (for example {@link Integer} or auto-boxed {@code int}),
* any {@link Token}, or {@link List}s of any of these.
*
*
* If you require a scope that is transparent or uses a different restriction, consider
* using one of the other provided scopes: {@link #scope}, {@link #transparentScope},
* {@link #nameScope}, or {@link #setFuelCostScope}. A "scope-transparency-matrix" can
* also be found in the interface comment for {@link Template}.
*
*
* Keeping hashtag-replacements local but letting {@link DataName}s escape can be
* useful in cases like the following, where we may want to reuse the hashtag
* multiple times:
*
*
* {@snippet lang=java :
* var template = Template.make(() -> scope(
* List.of("a", "b", "c").stream().map(name -> hashtagScope(
* let("name", name), // assumes values: a, b, c
* addDataName(name, PrimitiveType.INTS, MUTABLE), // escapes
* """
* int #name = 42;
* """
* ))
* // We still have access to the three DataNames.
* ));
* }
*
* @param tokens A list of tokens, which can be {@link String}s, boxed primitive types
* (for example {@link Integer}), any {@link Token}, or {@link List}s
* of any of these.
* @return The {@link ScopeToken} which captures the list of validated {@link Token}s.
* @throws IllegalArgumentException if the list of tokens contains an unexpected object.
*/
static ScopeToken hashtagScope(Object... tokens) {
return new ScopeTokenImpl(TokenParser.parse(tokens), true, false, true);
}
/**
* Creates a {@link ScopeToken} that represents a scope that is non-transparent for
* {@link #setFuelCost} (i.e. cannot escape), but transparent for hashtag-replacements,
* {@link DataName}s and {@link StructuralName}s (i.e. available in outer scope).
* The scope is formed from a list of tokens, which can be {@link String}s,
* boxed primitive types (for example {@link Integer} or auto-boxed {@code int}),
* any {@link Token}, or {@link List}s of any of these.
*
*
* If you require a scope that is transparent or uses a different restriction, consider
* using one of the other provided scopes: {@link #scope}, {@link #transparentScope},
* {@link #hashtagScope}, or {@link #nameScope}. A "scope-transparency-matrix" can
* also be found in the interface comment for {@link Template}.
*
*
* In some cases, it can be helpful to have different {@link #setFuelCost} within
* a single template, depending on the code nesting depth. Example:
*
*
* {@snippet lang=java :
* var template = Template.make(() -> scope(
* setFuelCost(1),
* // CODE1: some shallow code, allowing recursive template uses here
* // to use more fuel.
* """
* for (int i = 0; i < 1000; i++) {
* """,
* setFuelCostScope(
* setFuelCost(100)
* // CODE2: with the for-loop, we already have a deeper nesting
* // depth, and recursive template uses should not get
* // as much fuel as in CODE1.
* ),
* """
* }
* """
* // CODE3: we are back in the outer scope of CODE1, and can use
* // more fuel again in nested template uses. setFuelCost
* // is automatically restored to what was set before the
* // inner scope.
* ));
* }
*
* @param tokens A list of tokens, which can be {@link String}s, boxed primitive types
* (for example {@link Integer}), any {@link Token}, or {@link List}s
* of any of these.
* @return The {@link ScopeToken} which captures the list of validated {@link Token}s.
* @throws IllegalArgumentException if the list of tokens contains an unexpected object.
*/
static ScopeToken setFuelCostScope(Object... tokens) {
return new ScopeTokenImpl(TokenParser.parse(tokens), true, true, false);
}
/**
* Retrieves the dollar replacement of the {@code 'name'} for the
* current Template that is being instantiated. It returns the same
* dollar replacement as the string use {@code "$name"}.
*
*
* Here is an example where a Template creates a local variable {@code 'var'},
* with an implicit dollar replacement, and then captures that dollar replacement
* using {@link #$} for the use inside a nested template.
* {@snippet lang=java :
* var template = Template.make(() -> scope(
* """
* int $var = 42;
* """,
* otherTemplate.asToken($("var"))
* ));
* }
*
* @param name The {@link String} name of the name.
* @return The dollar replacement for the {@code 'name'}.
*/
static String $(String name) {
// Note, since the dollar replacements do not change within a template
// and the retrieval has no side effects, we can return the value immediately,
// and do not need a token.
return Renderer.getCurrent().$(name);
}
/**
* Define a hashtag replacement for {@code "#key"}, with a specific value.
*
*
* {@snippet lang=java :
* var template = Template.make("a", (Integer a) -> scope(
* let("b", a * 5),
* """
* System.out.println("Use a and b with hashtag replacement: #a and #b");
* """
* ));
* }
*
*
* Note that a {@code let} definition makes the hashtag replacement available
* for anything that follows it, until the the end of the next outer scope
* that is non-transparent for hashtag replacements. Additionally, hashtag
* replacements are limited to the template they were defined in.
* If you want to pass values from an outer to an inner template, this cannot
* be done with hashtags directly. Instead, one has to pass the values via
* template arguments.
*
* @param key Name for the hashtag replacement.
* @param value The value that the hashtag is replaced with.
* @return A token that represents the hashtag replacement definition.
*/
static Token let(String key, Object value) {
return new LetToken(key, value, v -> transparentScope());
}
/**
* Define a hashtag replacement for {@code "#key"}, with a specific value, which is also captured
* by the provided {@code function} with type {@code
* {@snippet lang=java :
* var template = Template.make("a", (Integer a) -> scope(
* let("b", a * 2, (Integer b) -> scope(
* """
* System.out.println("Use a and b with hashtag replacement: #a and #b");
* """,
* "System.out.println(\"Use a and b as capture variables:\"" + a + " and " + b + ");\n"
* ))
* ));
* }
*
* @param key Name for the hashtag replacement.
* @param value The value that the hashtag is replaced with.
* @param
* Example of a recursive Template, which checks the remaining {@link #fuel} at every level,
* and terminates if it reaches zero. It also demonstrates the use of {@link TemplateBinding} for
* the recursive use of Templates. We {@link Template.OneArg#render} with {@code 30} total fuel,
* and spend {@code 5} fuel at each recursion level.
*
*
* {@snippet lang=java :
* var binding = new TemplateBinding
*
*
*
*
* hashtag {@link DataName} and {@link StructuralName} {@link #setFuelCost}
*
*
* {@link #scope} non-transparent non-transparent non-transparent
*
*
* {@link #hashtagScope} non-transparent transparent transparent
*
*
* {@link #nameScope} transparent non-transparent transparent
*
*
* {@link #setFuelCostScope} transparent transparent non-transparent
*
*
* {@link #transparentScope} transparent transparent transparent
*