+ public enum TerminationStrategy {
+ /**
+ * Preserve token that is identified within Terminator and return it for processing. For example when
+ * building language parser, it could be used for statements that you want to capture.
+ */
+ PRESERVE,
+
+ /**
+ * While tokens that are marked by Terminator are identified, they are dropped and not returned for consumption.
+ * For example, when building language parser, you might use such strategy for whitespace and comments.
+ * That is, those tokens act as separators between actually useful tokens, but you don't want to consume such
+ * separators or comments in your code.
+ */
+ DROP
+ }