From: Svjatoslav Agejenko Date: Sat, 10 Jan 2026 13:15:36 +0000 (+0200) Subject: Add support for `finalAnswerIndicator` parameter to separate AI responses into intern... X-Git-Url: http://www2.svjatoslav.eu/gitweb/?a=commitdiff_plain;h=3d62e517f78cc2199d4692256812e1ea907627e8;p=alyverkko-cli.git Add support for `finalAnswerIndicator` parameter to separate AI responses into internal thoughts and final answer segments. --- diff --git a/doc/index.org b/doc/index.org index 71bce0c..795e9e9 100644 --- a/doc/index.org +++ b/doc/index.org @@ -693,20 +693,6 @@ Unlike *thread_count* which handles token generation, this phase is typically compute-bound rather than RAM-bound, so higher values often help up to your CPU's logical core count. -*** End of Text Marker - -An /end of text marker/ is an optional string (e.g., "###", ”[end of -text]“) specified per-model that signals the AI has completed its -response. When configured, Älyverkko CLI automatically truncates -output at this marker, removing any trailing artifacts. - -This parameter is useful with models that use specific termination -sequences so that they will not be shown to the AI user. - -For example, if a model typically ends responses with "###", setting -=end_of_text_marker: "###"= ensures the system removes "###" at the -end of AI response. - *** Context Size Tokens /Context size tokens/ defines the maximum number of tokens @@ -750,6 +736,78 @@ The system automatically selects the most specific applicable value, creating a flexible "rule cascade" where specialized configurations override broader ones. +** AI response post processing parameters +*** Final answer indicator +:PROPERTIES: +:ID: 8da2522b-d34f-4632-9fdd-603c1a64febb +:END: + +The *final_answer_indicator* is a optional, [[id:4206c6e9-d116-4030-94a4-87bf6f82043f][model-specific]] +configuration parameter that enables automatic separation of an AI's +response into two distinct sections: +- INTERNAL THOUGHTS :: The AI's reasoning process, step-by-step + analysis, and intermediate calculations +- ASSISTANT :: The concise final answer or conclusion + +This way, user can easily skip to final answer without reading through +lengthy reasoning. It is good for scenario where the reasoning process +is detailed but the conclusion is what matters most. + +This is only useful for *thinking* LLMs that produce internal thought +monologue before producing final response. Parameter is model specific +because different thinking models can use different ways to signal end +of internal thought and transition to final response mode. + +*How It Works:* +1. You define a unique string marker (e.g., =""=) in + your model's configuration: + + #+begin_src yaml + models: + - alias: "mistral" + filesystem_path: "Mistral-Large-Instruct-2407.Q8_0.gguf" + context_size_tokens: 32768 + final_answer_indicator: "" + #+end_src + + +2. When the AI generates a response: + - The system searches for the exact marker string in the response + - If found, it splits the response into two parts: + - Everything *before* the marker → =INTERNAL THOUGHTS= + - Everything *after* the marker → =ASSISTANT= + - If the marker is *not found*, the entire response appears in + =ASSISTANT= (fallback behavior) + +3. *Output Formatting*: The processed response is saved in your task + file with this structure: + + #+begin_example + DONE: skill=default model=mistral duration=38m + ,* USER: + [Original question] + + ,* INTERNAL THOUGHTS: + [AI's reasoning process] + + ,* ASSISTANT: + [Final answer] + #+end_example + +*** End of text marker + +An /end of text marker/ is an optional string (e.g., "###", ”[end of +text]“) specified per-model that signals the AI has completed its +response. When configured, Älyverkko CLI automatically truncates +output at this marker, removing any trailing artifacts. + +This parameter is useful with models that use specific termination +sequences so that they will not be shown to the AI user. + +For example, if a model typically ends responses with "###", setting +=end_of_text_marker: "###"= ensures the system removes "###" at the +end of AI response. + * Installation When you first encounter Älyverkko CLI, the setup process might seem @@ -919,6 +977,9 @@ Configuration file should be placed under current user home directory: bottleneck here. **** Model-Specific Settings +:PROPERTIES: +:ID: 4206c6e9-d116-4030-94a4-87bf6f82043f +:END: Each model in the =models= list can have: @@ -946,6 +1007,10 @@ Each model in the =models= list can have: can identify and remove them so that they don't leak into conversation. Default value is: *null*. +- =final_answer_indicator=: (Optional) Marker that allows to separate + thinking LLM internal thought from final response. Read more: [[id:8da2522b-d34f-4632-9fdd-603c1a64febb][Final + answer indicator]]. + *** Configuration file example The application is configured using a YAML-formatted configuration diff --git a/src/main/java/eu/svjatoslav/alyverkko_cli/commands/task_processor/TaskProcessorCommand.java b/src/main/java/eu/svjatoslav/alyverkko_cli/commands/task_processor/TaskProcessorCommand.java index c7ac0d5..57fd642 100644 --- a/src/main/java/eu/svjatoslav/alyverkko_cli/commands/task_processor/TaskProcessorCommand.java +++ b/src/main/java/eu/svjatoslav/alyverkko_cli/commands/task_processor/TaskProcessorCommand.java @@ -81,7 +81,7 @@ public class TaskProcessorCommand implements Command { } /** - * @return the name of this command, i.e., "mail". + * @return the name of this command, i.e., "process". */ @Override public String getCommandName() { @@ -222,11 +222,28 @@ public class TaskProcessorCommand implements Command { // Append the original user prompt (after the first line) resultFileContent.append(task.userPrompt).append("\n"); - // Append the AI response block - resultFileContent - .append("* ASSISTANT:\n") - .append(aiResponse) - .append("\n"); + // Check for final answer indicator in the model's configuration + String finalAnswerIndicator = task.model.finalAnswerIndicator; + if (finalAnswerIndicator != null && !finalAnswerIndicator.isEmpty()) { + int index = aiResponse.indexOf(finalAnswerIndicator); + if (index != -1) { + String internalThoughts = aiResponse.substring(0, index); + String finalAnswerPart = aiResponse.substring(index + finalAnswerIndicator.length()); + + resultFileContent.append("* INTERNAL THOUGHTS:\n"); + resultFileContent.append(internalThoughts).append("\n"); + + resultFileContent.append("* ASSISTANT:\n"); + resultFileContent.append(finalAnswerPart).append("\n"); + } else { + // If the indicator is present in model config but not in response, just put entire response in ASSISTANT + resultFileContent.append("* ASSISTANT:\n"); + resultFileContent.append(aiResponse).append("\n"); + } + } else { + resultFileContent.append("* ASSISTANT:\n"); + resultFileContent.append(aiResponse).append("\n"); + } File newFile = new File(file.getParentFile(), "DONE: " + file.getName()); saveToFile(newFile, resultFileContent.toString()); diff --git a/src/main/java/eu/svjatoslav/alyverkko_cli/configuration/ConfigurationModel.java b/src/main/java/eu/svjatoslav/alyverkko_cli/configuration/ConfigurationModel.java index 9588068..f330839 100644 --- a/src/main/java/eu/svjatoslav/alyverkko_cli/configuration/ConfigurationModel.java +++ b/src/main/java/eu/svjatoslav/alyverkko_cli/configuration/ConfigurationModel.java @@ -3,6 +3,7 @@ package eu.svjatoslav.alyverkko_cli.configuration; import com.fasterxml.jackson.annotation.JsonProperty; import lombok.Data; + /** * Represents a single AI model configuration entry, including alias, * path to the model file, token context size, and an optional @@ -39,7 +40,6 @@ public class ConfigurationModel { @JsonProperty("repeat_penalty") private Float repeatPenalty; - /** * The path to the model file (GGUF, etc.), relative to * {@link Configuration#getModelsDirectory()} or fully qualified. @@ -59,4 +59,12 @@ public class ConfigurationModel { */ @JsonProperty("end_of_text_marker") private String endOfTextMarker; + + /** + * Optional string that indicates the start of the final answer in the model's output. + * When specified, the response is split into ASSISTANT and FINAL ANSWER sections based on this indicator. + */ + @JsonProperty("final_answer_indicator") + private String finalAnswerIndicator; + } diff --git a/src/main/java/eu/svjatoslav/alyverkko_cli/model/Model.java b/src/main/java/eu/svjatoslav/alyverkko_cli/model/Model.java index 935ccd6..bac0aad 100644 --- a/src/main/java/eu/svjatoslav/alyverkko_cli/model/Model.java +++ b/src/main/java/eu/svjatoslav/alyverkko_cli/model/Model.java @@ -2,6 +2,7 @@ package eu.svjatoslav.alyverkko_cli.model; import java.io.File; + /** *

Represents an AI model stored on the filesystem with metadata about its capabilities and identification. * This class serves as a lightweight container for model information, enabling quick lookup and validation. @@ -62,7 +63,13 @@ public class Model { */ public Float repeatPenalty; - /** + /** + * Optional string indicating where the final answer starts in the model's response. + * If present in the response, the output is split into ASSISTANT (before indicator) and FINAL ANSWER (after indicator) sections. + */ + public String finalAnswerIndicator; + + /** * Constructs a {@link Model} instance with all hyperparameters. * * @param filesystemPath The path to the model file on the filesystem. @@ -74,11 +81,12 @@ public class Model { * @param repeatPenalty Penalty for token repetition (>0.0) * @param topK Token selection cutoff for Top-K sampling (>=1) * @param minP Minimum relative probability threshold (0.0-1.0) + * @param finalAnswerIndicator Optional string indicating where the final answer starts in the response. */ public Model(File filesystemPath, int contextSizeTokens, String modelAlias, String endOfTextMarker, Float temperature, Float topP, Float repeatPenalty, - Float topK, Float minP) { + Float topK, Float minP, String finalAnswerIndicator) { this.filesystemPath = filesystemPath; this.contextSizeTokens = contextSizeTokens; this.alias = modelAlias; @@ -88,6 +96,7 @@ public class Model { this.repeatPenalty = repeatPenalty; this.topK = topK; this.minP = minP; + this.finalAnswerIndicator = finalAnswerIndicator; } /** diff --git a/src/main/java/eu/svjatoslav/alyverkko_cli/model/ModelLibrary.java b/src/main/java/eu/svjatoslav/alyverkko_cli/model/ModelLibrary.java index 4c94c92..eb1087d 100644 --- a/src/main/java/eu/svjatoslav/alyverkko_cli/model/ModelLibrary.java +++ b/src/main/java/eu/svjatoslav/alyverkko_cli/model/ModelLibrary.java @@ -73,7 +73,8 @@ public class ModelLibrary { configModel.getRepeatPenalty(), configModel.getTopK(), - configModel.getMinP() + configModel.getMinP(), + configModel.getFinalAnswerIndicator() )); }