Skip to content

Shell Interpreter

Jump to bottom
opencode-agent[bot] edited this page Sep 27, 2026 · 1 revision

Shell Interpreter

JNode's line-oriented command interpreters: tokenizing, &&/||/; command sequences, pipelines, redirection, and the shared escapeWord round-trip.

Overview

JNode's shell has three independent interpreters. CommandShell picks one at startup and exposes it through the CommandInterpreter extension point:

  • DefaultInterpreter — single commands only. No pipelines, no redirection.
  • RedirectingInterpreter extends DefaultInterpreter — adds pipelines (|) and file redirection (<, >). This is the shell's initial interpreter.
  • BjorneInterpreter — a Bourne-compatible script interpreter with its own tokenizer, AST, and ListCommandNode; it has supported &&/||/; since long before the Java interpreters did.
// shell/src/shell/org/jnode/shell/CommandShell.java:114-116
public static final String INITIAL_INTERPRETER = "redirecting";
public static final String FALLBACK_INTERPRETER = "default";

DefaultInterpreter and RedirectingInterpreter share a single tokenizer, which lives as a nested class inside DefaultInterpreter. That is why a change to the tokenizer's operator rules immediately changes pipeline behavior too, even though the two interpreters have separate parsers.

This page covers the two Java interpreters. The argument-level parsing that runs after an interpreter has produced a CommandLine is a separate subsystem documented in Syntax and Shell-Syntax.

Key Components

Class / File Role
shell/src/shell/org/jnode/shell/CommandShell.java Interactive loop; INITIAL_INTERPRETER = "redirecting", FALLBACK_INTERPRETER = "default"
shell/src/shell/org/jnode/shell/DefaultInterpreter.java Tokenizer, parseSequence, interpret, escapeWord, completion (parsePartial)
shell/src/shell/org/jnode/shell/RedirectingInterpreter.java Two-level grammar: sequence of pipelines, each a list of commands with </>
shell/src/shell/org/jnode/shell/BjorneInterpreter.java Independent Bourne-compatible script interpreter
shell/src/shell/org/jnode/shell/syntax/SyntaxManager.java Validates the CommandLine against the command's declared syntax (see Syntax)
shell/src/test/org/jnode/test/shell/syntax/DefaultInterpreterSequenceTest.java 19 tests pinning sequence semantics, exit codes, and error propagation
shell/src/test/org/jnode/test/shell/syntax/DefaultTokenizerTest.java 20+ tests pinning token types for operator/quote/escape combinations

The Grammar

RedirectingInterpreter parses a two-level grammar; DefaultInterpreter implements only the outer level.

line    := sequence  EOF
sequence:= pipeline ( ('&&' | '||' | ';') pipeline )*
pipeline:= command ( '|' command )*          # RedirectingInterpreter only
command := name arg* ( '<' file | '>' file )*

Operators are left-associative by construction: both interpreters produce a flat List<...Entry>, not a precedence tree, and the evaluator walks it in order. There is no operator precedence between &&, || and ;.

DefaultInterpreter explicitly does not treat | as an operator — a bare | is a literal character in that interpreter.

Operator Constants

// DefaultInterpreter.java:81-89
public static final char AMP_CHAR = '&';
public static final char SEMI_CHAR = ';';
public static final String AND_IF = "&&";
public static final String OR_IF  = "||";
public static final String SEMI   = ";";
public static final String AMP    = "&";

Parsed Representation

// DefaultInterpreter.java:323-331
protected static class SequenceEntry {
    public final CommandLine commandLine;
    public final String operatorBefore;   // null for the first command
}

RedirectingInterpreter uses a parallel shape because its iteration unit is a pipeline, not a command:

// RedirectingInterpreter.java:535-546
private static class SequenceDescriptor {
    List<CommandDescriptor> pipeline;
    String operatorBefore;
}

Evaluation and Exit Status

DefaultInterpreter.interpret (DefaultInterpreter.java:194-233):

if (sequence.isEmpty()) return 0;
if (sequence.size() == 1) {
    return shell.invoke(sequence.get(0).commandLine, null, null);   // exceptions propagate
}
int rc = 0;
for (int i = 0; i < sequence.size(); i++) {
    SequenceEntry entry = sequence.get(i);
    if (i > 0) {
        String op = entry.operatorBefore;
        if (AND_IF.equals(op) && rc != 0) continue;      // skip
        else if (OR_IF.equals(op) && rc == 0) continue;  // skip
        // ';' always executes
    }
    try {
        rc = shell.invoke(entry.commandLine, null, null);
    } catch (ShellControlException ex) {
        throw ex;                                          // always rethrown
    } catch (ShellException ex) {
        shell.diagnose(ex, null);
        rc = 1;
    }
}
return rc;

Key semantics:

Rule Behavior
Single command Short-circuits to a bare shell.invoke, so ShellException (unknown command, script abort, test-harness trapException expectations) still propagates
Multiple commands Per-command ShellException is swallowed → shell.diagnose() + rc = 1; the sequence continues
ShellControlException Abstract subclass of ShellException; always rethrown even mid-sequence (used by TimeCommand and script abort)
Skipped command Does not reset rc, so false && true yields 1 and true || false yields 0 (POSIX-correct)
Return value The exit code of the last executed command, not the last parsed
$? Not available. The default interpreters only return rc up the call stack; there is no lastReturnCode equivalent to BjorneContext.getLastReturnCode()

RedirectingInterpreter.interpret has the identical shape but delegates each iteration to runSequence(shell, seq.pipeline), where runSequence chooses runCommand for a single command and runPipeline for two or more.

Tokenizer Rules

DefaultInterpreter.Tokenizer.tokenize (DefaultInterpreter.java:499-683) produces tokens of type LITERAL=0, STRING=1, CLOSED=2, SPECIAL=4. REDIRECTS_FLAG = 0x01 switches | from literal to operator.

Input Tokenizing Note
a && b, a || b, a ; b &&/||/; as SPECIAL recognized regardless of REDIRECTS_FLAG
a|b (no flag) one LITERAL a|b escaping beats the || operator rule
'a&&b', "c||d", 'e;f' STRING token quoting makes operator chars literal
a\&\&b, a||b, a\;b one LITERAL backslash escape
a | b (no flag) | is LITERAL only RedirectingInterpreter sets the flag
a | b (with flag) | is SPECIAL pipeline
http://x/?a=1&b=2 one LITERAL embedded single & stays literal (URL compatibility)
echo a & echo b & is SPECIAL standalone, so the parser can diagnose it
a &&& b && SPECIAL, then & SPECIAL parser error
a && b # c && d comment truncates the line tokenizer still emits a zero-length trailing LITERAL

The tokenizer is a nested class of DefaultInterpreter, so RedirectingInterpreter reaches it as DefaultInterpreter.Tokenizer and constructs it with REDIRECTS_FLAG.

escapeWord and the Round-Trip

CommandShell.escapeWord(String) must invert tokenization so that a command's output can be fed back into the shell. Since &, ; and | can now introduce operators, all three are escaped unconditionally:

// DefaultInterpreter.java:371-379 (escapeWord(String, boolean))
case AMP_CHAR:
case SEMI_CHAR:
case PIPE_CHAR:
    // '&', ';' and '|' can introduce operators ...
    sb.append(ESCAPE_CHAR).append(ch);
    break;

Previously | was only escaped when escapeRedirects == true, which is the path used by RedirectingInterpreter.escapeWord at RedirectingInterpreter.java:166-168. DefaultInterpreter.escapeWord(String) delegates to escapeWord(word, false) (:334-336).

Callers: CommandShell.escapeWord (:960), CommandCompletions (:79), BjorneContext (:1066), and the black-box test harness harness/CommandTestRunner (:44, :46). DefaultInterpreterSequenceTest.testEscapeWordRoundTrip (:334-345) pins the round-trip.

Syntax Errors

All raised as ShellSyntaxException:

Message Raised when
Misplaced '<op>': expected a command name &&/||/; appears where a command name was expected
no command after '<op>' sequence operator at end of line
unsupported '&': use '&&' for conditional execution standalone & reaches the parser
unrecognized symbol: '<text>' any other SPECIAL token
no command after '|' trailing pipe (RedirectingInterpreter only)
no filename after '<'/'>' redirection with no target (RedirectingInterpreter only)
misplaced '<tok>' special token in an illegal position (RedirectingInterpreter only)
empty '<special>' file name empty redirection target (RedirectingInterpreter only)

Errors are suppressed while the parser runs in completing mode (tab completion): a trailing operator yields an empty CommandLine rather than an exception.

Tab Completion

DefaultInterpreter.parsePartial (DefaultInterpreter.java:147-172) has to double-tokenize because parseSequence consumes the whole line:

Tokenizer tokenizer = new Tokenizer(line);
List<SequenceEntry> sequence = parseSequence(tokenizer, true);
if (sequence.isEmpty()) return new CommandLine("", null);
tokenizer.seek(0);                    // re-scan from the beginning
CommandLine.Token last = null;
while (tokenizer.hasNext()) {
    CommandLine.Token t = tokenizer.next();
    if (t.text.length() > 0) last = t;   // skip the empty token a #comment leaves behind
}
if (last != null && last.tokenType == SPECIAL
        && (AND_IF.equals(last.text) || OR_IF.equals(last.text) || SEMI.equals(last.text))) {
    return new CommandLine("", null);     // completing after a trailing operator
}
CommandLine res = sequence.get(sequence.size() - 1).commandLine;
return res == null ? new CommandLine("", null) : res;

This requires SymbolSource.seek(int) (DefaultInterpreter.java:708-713). RedirectingInterpreter.parsePartial (:124-129) accumulates SequenceDescriptors instead, and getLastCommand(List<SequenceDescriptor>) (:155-163) walks backwards skipping empty pipelines — it is also used by help().

Gotchas & Non-Obvious Behavior

  • | is a literal in DefaultInterpreter. A bare | only becomes SPECIAL when the REDIRECTS_FLAG is set, which only RedirectingInterpreter does.
  • Embedded & stays literal for URL compatibility. Only a standalone & (token buffer empty) is special. This is deliberate and covered by DefaultTokenizerTest.testTokenizerUrlWithAmp.
  • & is recognized but rejected. The tokenizer must surface it as SPECIAL so interpret can produce a helpful message instead of silently treating it as a literal.
  • No operator precedence. a || b && c evaluates strictly left to right, not as a || (b && c).
  • A single command still throws. The sequence.size() == 1 short-circuit is load-bearing for the black-box test harness and for script abort; removing it changes observable behavior of testSingleUnknownCommandRethrows and testSingleCommandFailurePropagates.
  • setArgumentAnticipated is only set on the last command of a sequence (DefaultInterpreter.java:313, RedirectingInterpreter.java:288-289). Earlier entries never get it, so "argument expected" completion state is only meaningful for the final command.
  • A trailing #comment leaks an empty argument. The comment token is zero-length but still lands in tokenList, so the last command of a sequence gains an empty-string argument. parsePartial works around it; interpret does not. Pre-existing quirk, preserved.
  • Sequence operators are recognized even without REDIRECTS_FLAG. Only | is flag-gated.
  • Testability requires subclasses. parseSequence, Tokenizer and the entry types are protected/private, so DefaultInterpreterSequenceTest drives them through TestableDefault, TestableRedirecting and a StubShell extends TestShell.
  • No $?, no && in Bjorne-parsed $(( )) expressions. The arithmetic evaluator (BJorne-Arithmetic-Evaluator) has its own tokenizer and is unaffected.

Related Pages

  • Shell-Commands — Parent hub for the command framework, invokers, and built-in commands
  • Syntax — Argument-level parsing (SyntaxManager, MuParser, Argument types) that runs after interpret
  • Shell-Syntax — Declarative argument syntax framework and tab-completion model
  • BJorne-Arithmetic-Evaluator — The independent Bourne-compatible interpreter's expression evaluator
  • JNode-Serial-Skill — Driving the shell over the serial agent console; notes which operators the interpreter supports
  • Testing — The two host-JVM test tiers: JUnit AllTests.java and black-box XML specs via TestHarness

Home | index

Architecture

Subsystems

Development

Reference

Clone this wiki locally