typically compute-bound rather than RAM-bound, so higher values often
help up to your CPU's logical core count.
-*** End of Text Marker
-
-An /end of text marker/ is an optional string (e.g., "###", ”[end of
-text]“) specified per-model that signals the AI has completed its
-response. When configured, Älyverkko CLI automatically truncates
-output at this marker, removing any trailing artifacts.
-
-This parameter is useful with models that use specific termination
-sequences so that they will not be shown to the AI user.
-
-For example, if a model typically ends responses with "###", setting
-=end_of_text_marker: "###"= ensures the system removes "###" at the
-end of AI response.
-
*** Context Size Tokens
/Context size tokens/ defines the maximum number of tokens
creating a flexible "rule cascade" where specialized configurations
override broader ones.
+** AI response post processing parameters
+*** Final answer indicator
+:PROPERTIES:
+:ID: 8da2522b-d34f-4632-9fdd-603c1a64febb
+:END:
+
+The *final_answer_indicator* is a optional, [[id:4206c6e9-d116-4030-94a4-87bf6f82043f][model-specific]]
+configuration parameter that enables automatic separation of an AI's
+response into two distinct sections:
+- INTERNAL THOUGHTS :: The AI's reasoning process, step-by-step
+ analysis, and intermediate calculations
+- ASSISTANT :: The concise final answer or conclusion
+
+This way, user can easily skip to final answer without reading through
+lengthy reasoning. It is good for scenario where the reasoning process
+is detailed but the conclusion is what matters most.
+
+This is only useful for *thinking* LLMs that produce internal thought
+monologue before producing final response. Parameter is model specific
+because different thinking models can use different ways to signal end
+of internal thought and transition to final response mode.
+
+*How It Works:*
+1. You define a unique string marker (e.g., ="<final_answer>"=) in
+ your model's configuration:
+
+ #+begin_src yaml
+ models:
+ - alias: "mistral"
+ filesystem_path: "Mistral-Large-Instruct-2407.Q8_0.gguf"
+ context_size_tokens: 32768
+ final_answer_indicator: "<final_answer>"
+ #+end_src
+
+
+2. When the AI generates a response:
+ - The system searches for the exact marker string in the response
+ - If found, it splits the response into two parts:
+ - Everything *before* the marker → =INTERNAL THOUGHTS=
+ - Everything *after* the marker → =ASSISTANT=
+ - If the marker is *not found*, the entire response appears in
+ =ASSISTANT= (fallback behavior)
+
+3. *Output Formatting*: The processed response is saved in your task
+ file with this structure:
+
+ #+begin_example
+ DONE: skill=default model=mistral duration=38m
+ ,* USER:
+ [Original question]
+
+ ,* INTERNAL THOUGHTS:
+ [AI's reasoning process]
+
+ ,* ASSISTANT:
+ [Final answer]
+ #+end_example
+
+*** End of text marker
+
+An /end of text marker/ is an optional string (e.g., "###", ”[end of
+text]“) specified per-model that signals the AI has completed its
+response. When configured, Älyverkko CLI automatically truncates
+output at this marker, removing any trailing artifacts.
+
+This parameter is useful with models that use specific termination
+sequences so that they will not be shown to the AI user.
+
+For example, if a model typically ends responses with "###", setting
+=end_of_text_marker: "###"= ensures the system removes "###" at the
+end of AI response.
+
* Installation
When you first encounter Älyverkko CLI, the setup process might seem
bottleneck here.
**** Model-Specific Settings
+:PROPERTIES:
+:ID: 4206c6e9-d116-4030-94a4-87bf6f82043f
+:END:
Each model in the =models= list can have:
can identify and remove them so that they don't leak into
conversation. Default value is: *null*.
+- =final_answer_indicator=: (Optional) Marker that allows to separate
+ thinking LLM internal thought from final response. Read more: [[id:8da2522b-d34f-4632-9fdd-603c1a64febb][Final
+ answer indicator]].
+
*** Configuration file example
The application is configured using a YAML-formatted configuration
}
/**
- * @return the name of this command, i.e., "mail".
+ * @return the name of this command, i.e., "process".
*/
@Override
public String getCommandName() {
// Append the original user prompt (after the first line)
resultFileContent.append(task.userPrompt).append("\n");
- // Append the AI response block
- resultFileContent
- .append("* ASSISTANT:\n")
- .append(aiResponse)
- .append("\n");
+ // Check for final answer indicator in the model's configuration
+ String finalAnswerIndicator = task.model.finalAnswerIndicator;
+ if (finalAnswerIndicator != null && !finalAnswerIndicator.isEmpty()) {
+ int index = aiResponse.indexOf(finalAnswerIndicator);
+ if (index != -1) {
+ String internalThoughts = aiResponse.substring(0, index);
+ String finalAnswerPart = aiResponse.substring(index + finalAnswerIndicator.length());
+
+ resultFileContent.append("* INTERNAL THOUGHTS:\n");
+ resultFileContent.append(internalThoughts).append("\n");
+
+ resultFileContent.append("* ASSISTANT:\n");
+ resultFileContent.append(finalAnswerPart).append("\n");
+ } else {
+ // If the indicator is present in model config but not in response, just put entire response in ASSISTANT
+ resultFileContent.append("* ASSISTANT:\n");
+ resultFileContent.append(aiResponse).append("\n");
+ }
+ } else {
+ resultFileContent.append("* ASSISTANT:\n");
+ resultFileContent.append(aiResponse).append("\n");
+ }
File newFile = new File(file.getParentFile(), "DONE: " + file.getName());
saveToFile(newFile, resultFileContent.toString());
import com.fasterxml.jackson.annotation.JsonProperty;
import lombok.Data;
+
/**
* Represents a single AI model configuration entry, including alias,
* path to the model file, token context size, and an optional
@JsonProperty("repeat_penalty")
private Float repeatPenalty;
-
/**
* The path to the model file (GGUF, etc.), relative to
* {@link Configuration#getModelsDirectory()} or fully qualified.
*/
@JsonProperty("end_of_text_marker")
private String endOfTextMarker;
+
+ /**
+ * Optional string that indicates the start of the final answer in the model's output.
+ * When specified, the response is split into ASSISTANT and FINAL ANSWER sections based on this indicator.
+ */
+ @JsonProperty("final_answer_indicator")
+ private String finalAnswerIndicator;
+
}
import java.io.File;
+
/**
* <p>Represents an AI model stored on the filesystem with metadata about its capabilities and identification.
* This class serves as a lightweight container for model information, enabling quick lookup and validation.
*/
public Float repeatPenalty;
- /**
+ /**
+ * Optional string indicating where the final answer starts in the model's response.
+ * If present in the response, the output is split into ASSISTANT (before indicator) and FINAL ANSWER (after indicator) sections.
+ */
+ public String finalAnswerIndicator;
+
+ /**
* Constructs a {@link Model} instance with all hyperparameters.
*
* @param filesystemPath The path to the model file on the filesystem.
* @param repeatPenalty Penalty for token repetition (>0.0)
* @param topK Token selection cutoff for Top-K sampling (>=1)
* @param minP Minimum relative probability threshold (0.0-1.0)
+ * @param finalAnswerIndicator Optional string indicating where the final answer starts in the response.
*/
public Model(File filesystemPath, int contextSizeTokens,
String modelAlias, String endOfTextMarker,
Float temperature, Float topP, Float repeatPenalty,
- Float topK, Float minP) {
+ Float topK, Float minP, String finalAnswerIndicator) {
this.filesystemPath = filesystemPath;
this.contextSizeTokens = contextSizeTokens;
this.alias = modelAlias;
this.repeatPenalty = repeatPenalty;
this.topK = topK;
this.minP = minP;
+ this.finalAnswerIndicator = finalAnswerIndicator;
}
/**
projects / alyverkko-cli.git / commitdiff