1
0
Fork 0
mirror of git://git.code.sf.net/p/cdesktopenv/code synced 2025-03-09 15:50:02 +00:00
Commit graph

261 commits

Author SHA1 Message Date
Martijn Dekker
f8f2c4b608 Remove obsolete quote balancing hack
The old Bourne shell failed to check for closing quotes and command
substitution backticks when encountering end-of-file in a parser
context (such as a script). ksh93 implemented a hack for partial
compatibility with this bug, tolerating unbalanced quotes and
backticks in backtick command subsitutions, 'eval', and command
line invocation '-c' scripts only.

This hack became broken for backtick command substitutions in
fe20311f/350b52ea as a memory leak was fixed by adding a newline to
the stack at the end of the command substitution. That extra
newline becomes part of any string whose quotes are not properly
terminated, causing problems such as the one detailed here:
https://www.mail-archive.com/ast-developers@lists.research.att.com/msg01889.html

    $ touch abc
    $ echo `ls "abc`
    ls: abc
    : not found

No other fix for the memory leak is known that doesn't cause other
problems. (The alternative fix detailed in the referenced mailing
list post causes a different corner-case regression.)

Besides, the hack has always caused other corner case bugs as well:

	$ ksh -c '((i++'
Actual:	ksh: i++(: not found
	(If an external command 'i++(' existed, it would be run)
Expect:	ksh: syntax error at line 1: `(' unmatched

	$ ksh -c 'i=0; echo $((++i'
Actual:	(empty line; the arithmetic expansion is ignored)
Expect:	ksh: syntax error at line 1: `(' unmatched

	$ ksh -c 'echo $(echo "hi)'
Actual:	ksh: syntax error at line 1: `(' unmatched
Expect: ksh: syntax error at line 1: `"' unmatched

So, it's time to get rid of this hack. The old Bourne shell is
dead and buried. No other shell tries to support this breakage.
Tolerating syntax errors is just asking for strange side effects,
inconsistent states, and corner case bugs. We should not want to do
that. Old scripts that rely on this will just need to be fixed.

src/cmd/ksh93/sh/lex.c:
- struct lexdata: Remove 'char balance' member for remembering an
  unbalanced quote or backtick.
- sh_lex(): Remove the back to remember and compensate for
  unbalanced quotes/backticks that was executed only if we were
  executing a script from a string, as opposed to a file.

src/cmd/ksh93/COMPATIBILITY:
- Note the change.

Resolves: https://github.com/ksh93/ksh/issues/199
2021-03-05 22:17:14 +00:00
Martijn Dekker
b48e5b3365 Fix arbitrary command execution vuln in array subscripts in arith
This commit fixes an arbitrary command execution vulnerability in
array subscripts used within the arithmetic subsystem.

One of the possible reproducers is:
	var='1$(echo INJECTION >&2)' ksh -c \
		'typeset -A a; ((a[$var]++)); typeset -p a'

Output before this commit:
	INJECTION
	typeset -A a=([1]=1)
The 'echo' command has been surreptitiously executed from an
external environment variable.

Output after this commit:
	typeset -A a=(['1$(echo INJECTION >&2)']=1)
The value is correctly used as an array subscript and nothing in it
is parsed or executed. This is as it should be, as ksh93 supports
arbitrary subscripts for associative arrays.

If we think about it logically, the C-style arithmetic subsystem
simply has no business messing around with shell expansions or
quoting at all, because those don't belong to it. Shell expansions
and quotes are properly resolved by the main shell language before
the arithmetic subsystem is even invoked. It is particularly
important to maintain that separation because the shell expansion
mechanism also executes command substitutions.

Yet, the arithmetic subsystem subjected array subscripts that
contain `$` (and only array subscripts -- how oddly specific) to
an additional level of expansion and quote resolution. For some
unfathomable reason, there are two lines of code doing specifically
this. The vulnerability is fixed by simply removing those.

Incredibly, variants of this vulnerability are shared by bash, mksh
and zsh. Instead of fixing it, it got listed in Bash Pitfalls!
http://mywiki.wooledge.org/BashPitfalls#y.3D.24.28.28_array.5B.24x.5D_.29.29

src/cmd/ksh93/sh/arith.c:
- scope(): Remove these two lines that implement the vulnerability.
			if(strchr(sub,'$'))
				sub = sh_mactrim(shp,sub,0);
- scope(), arith(): Remove the NV_SUBQUOTE flag from two
  nv_endsubscript() calls. That flag causes the array subscript to
  retain the current level of shell quoting. The shell quotes
  everything as in "double quotes" before invoking the arithmetic
  subsystem, and the bad sh_mactrim() call removed one level of
  quoting. Since we're no longer doing that, this flag should no
  longer be passed, or subscripts may get extra backslash escapes.

src/cmd/ksh93/include/name.h,
src/cmd/ksh93/sh/array.c:
- nv_endsubscript(): The NV_SUBQUOTE flag was only passed from
  arith.c. Since it is now unused, remove it.

src/cmd/ksh93/tests/arith.sh:
- Tweak some tests: fix typos, report wrong values.
- Add 21 tests. Most are based on reproducers contributed by
  @stephane-chazelas and @hyenias. They verify that this
  vulnerability is gone and that no quoting bugs were introduced.

Resolves: https://github.com/ksh93/ksh/issues/152
2021-03-04 13:37:13 +00:00
Martijn Dekker
6146848693 Fix compiling with SHOPT_REGRESS and SHOPT_P_SUID
src/cmd/ksh93/Mamfile:
- regress.c: add missing SH_DICT define for getopt self-doc string,
  needed after USAGE_LICENSE macros were removed. (re: ede47996)

src/cmd/ksh93/init.c: sh_init():
- Do not set error_info.exit early in init. This is the function
  that is called when an error exits the shell. It defaults to
  exit(3). Setting it to sh_exit() early on can cause a crash if an
  error is thrown before shell initialisation is fully finished.
  So set it at the end of sh_init() instead.
- __regress__: Remove error_info.exit workaround. (re: 506bd2b2)
- Fix SHOPT_P_SUID directive. This is not actually a 0/1 value, so
  we should use #ifdef and not #if. If SHOPT_REGRESS is on, it it
  set to a function call. (re: 2182ecfa)

src/cmd/ksh93/SHOPT.sh:
- Document that SHOPT_P_SUID cannot be set to 0 to be turned off.
2021-02-28 23:24:58 +00:00
Johnothan King
7ad274f8b6
Add more out of memory checks (re: 18529b88) (#192)
The referenced commit neglected to add checks for strdup() calls.
That calls malloc() as well, and is used a lot.

This commit switches to another strategy: it adds wrapper functions
for all the allocation macros that check if the allocation
succeeded, so those checks don't need to be done manually.

src/cmd/ksh93/include/defs.h,
src/cmd/ksh93/sh/init.c:
- Add sh_malloc(), sh_realloc(), sh_calloc(), sh_strdup(),
  sh_memdup() wrapper functions with success checks. Call nospace()
  to error out if allocation fails.
- Update new_of() macro to use sh_malloc().
- Define new sh_newof() macro to replace newof(); it uses
  sh_realloc().

All other changed files:
- Replace the relevant calls with the wrappers.
- Remove now-redundant success checks from 18529b88.
- The ERROR_PANIC error message calls are updated to inclusive-or
  ERROR_SYSTEM into the exit code argument, so libast's error()
  appends the human-readable version of errno in square brackets.
  See src/lib/libast/man/error.3

src/cmd/ksh93/edit/history.c:
- Include "defs.h" to get access to the wrappers even if KSHELL is
  not defined.
- Since we're here, fix a compile error that occurred with KSHELL
  undefined by updating the type definition of hist_fname[] to
  match that of history.h.

src/cmd/ksh93/bltins/enum.c:
- To get access to sh_newof(), include "defs.h" instead of
  <shell.h> (note that "defs.h" includes <shell.h> itself).

src/cmd/ksh93/Mamfile:
- enum.c: depend on defs.h instead of shell.h.
- enum.o: add an -I. flag in the compiler invocation so that defs.h
  can find its subsequent includes.

src/cmd/builtin/pty.c:
- Define one outofmemory() function and call that instead of
  repeating the error message call.
- outofmemory() never returns, so remove superfluous exit handling.

Co-authored-by: Martijn Dekker <martijn@inlv.org>
2021-02-27 21:21:58 +00:00
Martijn Dekker
c928046aa9 Fix ${.sh.fun} leaking out of DEBUG trap
The value of the ${.sh.fun} variable, which is supposed to contain
the name of the function currently being executed, leaks out of the
DEBUG trap if it executes a function. Reproducer:

$ fn() { echo "executing the function"; }
$ trap fn DEBUG
$ trap - DEBUG
executing the function
$ echo ${.sh.fun}
fn

${.sh.fun} should be empty outside the function.

Annalysis:

The sh_debug() function in xec.c, which executes the DEBUG trap
action, contains these lines, which are part of restoring the state
after running the trap action with sh_trap():

	nv_putval(SH_PATHNAMENOD,shp->st.filename,NV_NOFREE);
	nv_putval(SH_FUNNAMENOD,shp->st.funname,NV_NOFREE);
 	shp->st = savst;

First the SH_PATHNAMENOD (${.sh.file}) and SH_FUNNAMENOD
(${.sh.fun}) variables get restored from the values in the shell's
scoped information struct (shp->st), but that is done *before*
restoring the parent scope with 'shp->st = savst;'. It should be
done after. Fixing the order is sufficient to fix the bug.

However, I am not convinced that these nv_putval() calls are good
for anything at all. Setting, unsetting, restoring, etc. the
${.sh.fun} and ${.sh.file} variables is already being handled
perfectly well elsewhere in the code for executing functions and
sourcing dot scripts. The DEBUG trap is neither here nor there.
There's no reason for it to get involved with these variables.

I was unable to break anything after simply removing those two
lines. So I strongly suspect this is another case, out of many now,
where a bug in ksh93 is properly fixed by removing some code.

I couldn't get ${.sh.file} to leak similarly -- I think this is
because SH_PATHNAMENOD (and not SH_FUNNOD) is set explicitly in
exfile() in main.c, masking this incorrect restore. It is the only
place where SH_PATHNAMENOD and SH_FUNNOD are not both set.

src/cmd/ksh93/sh/xec.c:
- Remove these two spurious nv_putval() calls.

src/cmd/ksh93/tests/variables.sh:
- Add regression test for leaking ${.sh.fun}.
2021-02-27 01:25:59 +00:00
Martijn Dekker
caf7ab6c71 Make PATH properly survive a shared-state ${ comsub; }
Reproducer:

$ ksh -c 'v=${ PATH=/dev/null; }; echo $PATH; whence ls'
/dev/null
/bin/ls

The PATH=/dev/null assignment should survive the shared-state
command substitution, and does, yet 'ls' is still found.
The variable became inconsistent with the internal pathlist.

This bugfix is from the 93v- beta.

src/cmd/ksh93/sh/subshell.c: sh_subshell():
- Do not save and restore pathlist for a subshare.
- A few other subshell tweaks from 93v- that made sense:
  . reset shp->subdup (bitmask for dups of 1) after saving it
  . use e_dot instead of "." for consistency
  . retry close(1) if it was interrupted

src/cmd/ksh93/tests/path.sh:
- Add test for this bug.
2021-02-23 22:16:06 +00:00
Johnothan King
733f70e94b
Fix many compiler warnings and remove unused variables (#191)
Most of these changes remove unused variables, functions and labels
to fix -Wunused compiler warnings. Somewhat notable changes:

src/cmd/ksh93/bltins/print.c:
- Removed the unused 'neg' variable.
  Patch from ksh2020: https://github.com/att/ast/pull/725

src/cmd/ksh93/bltins/sleep.c:
- Initialized ns to fix three -Wsometimes-uninitialized warnings.

src/cmd/ksh93/edit/{emacs,vi}.c:
- Adjust strncpy size to fix two -Wstringop-truncation warnings.

src/cmd/ksh93/include/shell.h:
- The NOT_USED macro caused many -Wunused-value warnings,
  so it has been replaced with ksh2020's macro:
  19d0620a

src/cmd/ksh93/sh/expand.c:
- Removed an unnecessary 'ap = ' since 'ap' is never read
  between stakseek and stakfreeze.

src/cmd/ksh93/edit/vi.c: refresh():
- Undef this function's 'w' macro at the end of it to stop it
  potentially interfering with future code changes.

src/cmd/ksh93/sh/nvdisc.c,
src/lib/libast/misc/magic.c,
src/lib/libast/regex/regsubexec.c,
src/lib/libast/sfio/sfpool.c,
src/lib/libast/vmalloc/vmbest.c:
- Fixed some indentation to silence -Wmisleading-indentation
  warnings.

src/lib/libast/include/ast.h:
- For clang, now only suppress hundreds of -Wparentheses warnings
  as well as a few -Wstring-plus-int warnings.
  Clang's -Wparentheses warns about things like
  	if(foo = bar())
  which assigns to foo and checks the assigned value.
  Clang wants us to change this into
  	if((foo = bar()))
  Clang's -Wstring-plus-int warns about things like
  	"string"+x
  where x is an integer, e.g. "string"+3 represents the string
  "ing". Clang wants us to change that to
  	"string"[3]
  The original versions represent a perfectly valid coding style
  that was common in the 1980s and 1990s and is not going to change
  in this historic code base. (gcc does not complain about these.)

Co-authored-by: Martijn Dekker <martijn@inlv.org>
2021-02-22 22:16:32 +00:00
Martijn Dekker
18529b88c6 Add lots of checks for out of memory (re: 0ce0b671)
Huge typeset -L/-R adjustment length values were still causing
crashses on sytems with not enough memory. They should error out
gracefully instead of crashing.

This commit adds out of memory checks to all malloc/calloc/realloc
calls that didn't have them (which is all but two or three).

The stkalloc/stakalloc calls don't need the checks; it has
automatic checking, which is done by passing a pointer to the
outofspace() function to the stakinstall() call in init.c.

src/lib/libast/include/error.h:
- Change the ERROR_PANIC exit status value from ERROR_LEVEL (255)
  to 77, which is what it is supposed to be according to the libast
  error.3 manual page. Exit statuses > 128 for anything else than
  signals are not POSIX compliant and may cause misbehaviour.

src/cmd/ksh93/include/defs.h,
src/cmd/ksh93/sh/init.c:
- To facilitate consistency, add a simple extern sh_outofmemory()
  function that throws an ERROR_PANIC "out of memory".

src/cmd/ksh93/include/shell.h,
src/cmd/ksh93/data/builtins.c:
- Remove now-redundant e_nospace[] extern message; it is now only
  used in one place so it might as well be a string literal in
  sh_outofmemory().

All other changed files:
- Verify the result of all malloc/calloc/realloc calls and call
  sh_outofmemory() if they fail.
2021-02-21 22:27:28 +00:00
hyenias
0ce0b67149
Fix segmentation fault for justified strings (re: bdb99741) (#190)
Additional adjustments to previous commit bdb9974 to correct
crashes when the max size of a justified string is requested.
This commit corrects the following:

Before (Ubuntu 64bit):
$ typeset -L $(((1<<31)-1)) s=h; typeset +p s
Segmentation fault (core dumped)

After:
$ typeset -L $(((1<<31)-1)) s=h; typeset +p s
typeset -L 2147483647 s

src/cmd/ksh93/sh/name.c: nv_putval():
- Alter the variables size, dot, and append from int to unsigned
  int to prevent unwanted negative values from being expressed.
- By creating size, dot, and append as unsigned ints; (unsigned)
  type casting is avoided.
2021-02-21 09:34:18 +00:00
Martijn Dekker
51b2e360fa job_reap(): fix use of unitialised pointer
This solves another intermittent crash that happened upon
processing SIGWINCH in the emacs editor. See also: 7ff6b73b

I found this bug while testing ksh 93u+m on OpenBSD. Due to its
pervasive security hardening, this system crashes a program
reliably where others crash it intermittently, which is invaluable.

src/cmd/ksh93/sh/jobs.c: job_reap():
- The pw pointer is not ever given a value if the loop breaks on
  line 318-319, but it is used unconditionally on lines 464-470,
  Initialise the pointer to null on function entry and do not call
  job_list() and job_unpost() if the pointer is still null.
2021-02-20 23:40:00 +00:00
Martijn Dekker
500757d78b Error out on 'redirect >foo' inside ${ shared-state comsub; }
The following caused an infinite loop:

        v=${ exec >/dev/tty; }
        v=${ redirect >/dev/tty; }

Even the original authors didn't figure out how to 'exec >foo' or
'redirect >foo' inside a non-forking command substitution, so they
fork it by calling sh_subfork(). If we delete that call, even
normal command substitutions enter into that infinite loop. But of
course a shared-state comsub can never fork as it would no longer
share its state. Without a solution to make this work without
forking, an error message is the only sensible thing left to do.

src/cmd/ksh93/sh/io.c: sh_redirect():
- If we're redirecting standard output (1), the redirection is
  permanent as in 'exec'/'redirect' (flag==2), and we're in a
  subshare, then error out.

Resolves: https://github.com/ksh93/ksh/issues/128
2021-02-20 19:52:08 +00:00
Martijn Dekker
bdb997415d Fix multiple buffer overflows with justified strings (-L/-R/-Z)
ksh crashed in various different and operating system-dependent
ways when attempting to create or apply justification strings
using typeset -L/-R/-Z, especially if large sizes are used.

The crashes had two immediate causes:
- In nv_newattr(), when applying justification attributes, a buffer
  was allocated for the justified string that was exactly 8 bytes
  longer than the original string. Any larger justification string
  caused a buffer overflow (!!!).
- In nv_putval(), when applying existing attributes to a new value,
  the corresponding memmove() either did not zero-terminate the
  justified string (if the original string was longer than the
  justified string) or could read memory past the original string
  (if the original string was shorter than the justified string).
  Both scenarios can cause a crash.

This commit fixes other minor issues as well, such as a mysterious
8 extra bytes allocated by several malloc/realloc calls. This may
have been some naive attempt to paper over the above bugs. It seems
no one can make any other kind of sense of it.

A readjustment bug with zero-filling was also fixed.

src/cmd/ksh93/sh/name.c:
- nv_putval():
  . Get rid of the magical +8 bytes for malloc and realloc. Just
    allocate one extra byte for the terminating zero.
  . Fix the memmove operation to use strncpy instead, so that
    buffer overflows are avoided in both scenarios described above.
    Also make it conditional upon a size adjustment actually
    happening (i.e. if 'dot' is nonzero).
  . Mild refactoring: combine two 'if(sp)' blocks into one;
    declare variables only used there locally for legibility.
- nv_newattr():
  * Replace the fatally broken "let's allocate string length + 8
    bytes no matter the size of the adjustment" routine with a new
    one based on work by @hyenias (see comments in #142). It is
    efficient with memory, taking into account numeric types,
    growing strings, and shrinking strings.
  * Fix zero-filling in readjustment after changing the initial
    size of a -Z attribute. If the number was zero, all zeros were
    still skipped, leaving an empty string.

Thanks to @hyenias for originally identifying this breakage and
laying the groundwork for fixing nv_newattr(), and to @lijog for
the crash analysis that revealed the key to the nv_putval() fix.

Resolves: https://github.com/ksh93/ksh/issues/142
Resolves: https://github.com/ksh93/ksh/issues/181
2021-02-20 13:05:38 +00:00
Martijn Dekker
a959a35291 DEBUG trap: restore status 2 trigger to skip command (re: d00b4b39)
So now we know what that faulty check for shp->indebug in sh_trap()
was meant to do: it was meant to pass down the trap handler's exit
status, via sh_debug(), down to sh_exec() (xec.c) so that it could
then skip the execution of the next command if the trap's exit
status is 2, as documented in the manual page. As of d00b4b39, exit
status 2 was not passed down, so this stopped working.

This commit reinstates that functionality, but without the exit
status bug in command substitutions caused by the old way.

src/cmd/ksh93/sh/fault.c: sh_trap():
- Save the trap's exit status before restoring the parent
  envionment's exit status. Make this saved exit status the return
  value of the function. (This does not break anything, AFAICT; the
  majority of sh_trap() calls ignore the return value, and the few
  that don't ignore it seem to expect it to return exactly this.)

src/cmd/ksh93/sh/xec.c: sh_exec():
- The sh_trap() fix has one side effect: whereas the exit status of
  a skipped command was always 2 (as per the trap handler), now it
  is always 0, because it gets reset in sh_exec() but no command is
  executed. That is probably not a desirable change in behaviour,
  so let's fix that here instead: set sh.exitval to 2 when skipping
  commands.

src/cmd/ksh93/sh.1:
- Document that ${.sh.command} shell-quotes its arguments for use
  by 'eval' and such. This fact was not documented anywhere, AFAIK.

src/cmd/ksh93/shell.3:
- Document that $? (exit status) is made local to trap handlers.
- Document that sh_trap() returns the trap handler's exit status.

src/cmd/ksh93/tests/basic.sh:
- Add test for this bug.
- Add a missing test for the exit status 255 functionality (if a
  DEBUG trap handler yields this exit status and we're executing a
  function or dot script, a return is triggered).

Fixes: https://github.com/ksh93/ksh/issues/187
2021-02-20 05:13:51 +00:00
Johnothan King
2b805f7f1c
Fix many spelling errors and word repetitions (#188)
Many of the errors fixed in this commit are word repetitions
such as 'the the' and minor spelling errors. One formatting
error in the ksh man page has also been fixed.
2021-02-20 03:22:24 +00:00
Martijn Dekker
350b52ea4e Update comsub-with-alias anti-leak hack (re: fe20311f)
In the 93v- beta, they add a newline instead of a space.
This has fewer side effects as final newlines get stripped.
It's still a hack and it would still be nice to have a real fix,
but it seems even the AT&T guys couldn't come up with one.

src/cmd/ksh93/sh/macro.c:
- To somehow avoid a memory leak involving alias substitution,
  append a linefeed instead of a space to the comsub buffer.

src/cmd/ksh93/tests/subshell.sh:
- Add test for minor regression caused by the RedHat version.
2021-02-18 23:35:20 +00:00
Martijn Dekker
8c2d8e5f46 Revert 096f46ee ("Fix for memory mgmt in variable expansion")
This reverts a Solaris patch (105-CR7032068) with no documentation
on what it fixes or how or why. There are reports about it causing
a crash and/or a regression:

https://github.com/ksh93/ksh/issues/180#issuecomment-780980442
2021-02-18 02:05:39 +00:00
Martijn Dekker
50b665b1ed Revert 0e4c4d61 ("Fix minor typeset attribute regressions")
This commit introduced the following bug, which is worse than the
one that commit fixed: it became impossible to alter the size of an
existing justified string attribute.

Thanks to @hyenias for catching this bug:
https://github.com/ksh93/ksh/issues/142#issuecomment-780931533

$ unset s; typeset -L 100 s=h; typeset +p s; typeset -L 5 s; typeset +p s
typeset -L 100 s
typeset -L 100 s

Expected output:
typeset -L 100 s
typeset -L 5 s

src/cmd/ksh93/sh/name.c:
- Revert.

src/cmd/ksh93/tests/attributes.sh:
- Revert: re-disable tests for minor attribute output regressions.
- Add a test for this bug and potential similar bugs.
2021-02-18 01:11:53 +00:00
Martijn Dekker
911d6b066f Fix subshell scoping of changes in shared command substitution
A ${ shared-state command substitution; } (internally called
subshare) is documented to share its state with the parent shell
environment, so all changes made within the command substitution
survive outside of it. However, when it is run within a
virtual/non-forked subshell, variables that are not already local
to that subshell will leak out of it into the grandparent state.
Reproducer:

	$ ksh -c '( v=${ bug=BAD; } ); echo "$bug"'
	BAD

If the variable pre-exists in the subshell, the bug does not occur:

	$ ksh -c '( bug=BAD1; v=${ bug=BAD2; } ); echo "$bug"'
	(empty line, as expected)

The problem is that the sh_assignok() function, which is
responsible for variable scoping in virtual subshells, does not
ever bother to create a virtual subshell scope for a subshare.
That is an error if a subshare's parent (or higher-up ancestor)
environment is a virtual subshell, because a scope needs to be
created in that parent environment if none exists.

To make this bugfix possible, first we need to get something out of
the way. nv_restore() temporarily sets the subshell's pointer to
the preesnt working directory, shpwd, to null. This causes
sh_assignok() to assume that the subshell is a subshare (because
subshares don't store their own PWD) and refuse to create a scope.
However, nv_restore() sets it to null for a different purpose: to
temporarily disable scoping for *all* virtual subshells, making
restoring possible. This is a good illustration of why it's often
not a good idea to use the same variable for unrelated purposes.

src/cmd/ksh93/sh/subshell.c:
- Add a global static subshell_noscope flag variable to replace the
  misuse of sh.shpwd described above.
- sh_assignok():
  . Check subshell_noscope instead of shpwd to see if scope
    creation is disabled. This makes it possible to distinguish
    between restoring scope and handling subshares.
  . If the current environment is a subshare that is in a virtual
    subshell, create a scope in the parent subshell. This is done
    by temporarily making the parent virtual subshell the current
    subshell (by setting the global subshell_data pointer to it)
    and calling sh_assignok() again, recursively.
- nv_restore(): To disable subshell scope creation while restoring,
  set subshell_noscope instead of saving and unsetting sh.shpwd.

src/cmd/ksh93/tests/subshell.sh:
- Add tests. I like tests. Tests are good.

Fixes: https://github.com/ksh93/ksh/issues/143
2021-02-17 15:33:48 +00:00
Martijn Dekker
e37aa358bf Fix BUG_CASEEMPT: empty 'case' list was syntax error
'case x in esac' should be syntactically correct, but was an error:

	$ ksh -c 'case x in esac'
	ksh: syntax error at line 1: `case' unmatched

Inserting a newline was a workaround:

	$ ksh -c $'case x in\nesac'
	(no output)

The problem was that the 'esac' reserved word was not being
recognised if it immediately followed the 'in' reserved word.

src/cmd/ksh93/sh/lex.c: sh_lex():
- Do not turn off recognition of reserved words after 'in' if we're
  in a 'case' construct; only do this for 'for' and 'select'.

src/cmd/ksh93/tests/case.sh:
- Add seven regression test for correct recognition of 'esac'.
  Only two failed on ksh93. The rest is to catch future bugs.

Fixes: https://github.com/ksh93/ksh/issues/177
2021-02-16 06:50:12 +00:00
Martijn Dekker
af5f7acf99 Fix bugs related to --posix shell option (re: 921bbcae, f45a0f16)
This fixes the following:
1. 'set --posix' now works as an equivalent of 'set -o posix'.
2. The posix option turns off braceexpand and turns on letoctal.
   Any attempt to override that in a single command such as 'set -o
   posix +o letoctal' was quietly ignored. This now works as long
   as the overriding option follows the posix option in the command.
3. The --default option to 'set' now stops the 'posix' option, if
   set or unset in the same 'set' command, from changing other
   options. This allows the command output by 'set +o' to correctly
   restore the current options.

src/cmd/ksh93/data/builtins.c:
- To make 'set --posix' work, we must explicitly list it in
  sh_set[] as a supported option so that AST optget(3) recognises
  it and won't override it with its own default --posix option,
  which converts the optget(3) string to at POSIX getopt(3) string.
    This means it will appear as a separate entry in --man output,
  whether we want it to or not. So we might as well use it as an
  example to document how --optionname == -o optionname, replacing
  the original documentation that was part of the '-o' description.

src/cmd/ksh93/sh/args.c: sh_argopts():
- Add handling for explitit --posix option in data/builtins.c.
- Move SH_POSIX syncing SH_BRACEEXPAND and SH_LETOCTAL from
  sh_applyopts() into the option parsing loop here. This fixes
  the bug that letoctal was ignored in 'set -o posix +o letoctal'.
- Remember if --default was used in a flag, and do not sync options
  with SH_POSIX if the flag is set. This makes 'set +o' work.

src/cmd/ksh93/include/argnod.h,
src/cmd/ksh93/data/msg.c,
src/cmd/ksh93/sh/args.c: sh_printopts():
- Do not potentially translate the 'on' and 'off' labels in 'set
  -o' output. No other shell does, and some scripts parse these.

src/cmd/ksh93/sh/init.c: sh_init():
- Turn on SH_LETOCTAL early along with SH_POSIX if the shell was
  invoked as sh; this makes 'sh -o' and 'sh +o' show expected
  options (not that anyone does this, but correctness is good).

src/cmd/ksh93/include/defs.h,
src/cmd/ksh93/include/shell.h:
- The state flags were in defs.h and most (but not all) of the
  shell options were in shell.h. Gather all the shell state and
  option flag definitions into one place in shell.h for clarity.
- Remove unused SH_NOPROFILE and SH_XARGS option flags.

src/cmd/ksh93/tests/options.sh:
- Add tests for these bugs.

src/lib/libast/misc/optget.c: styles[]:
- Edit default optget(3) option self-documentation for clarity.

Several changed files:
- Some SHOPT_PFSH fixes to avoid compiling dead code.
2021-02-14 23:51:19 +00:00
Martijn Dekker
fb7551634a Fix exec/redirect (fd != 1) in shared-state comsub (re: db72f41f)
That OpenSUSE patch introduced a bug: file descriptors other than 1
that were globally redirected using 'exec' or 'redirect' no longer
survived a ${ shared-state; } command substitution.

Related: https://github.com/ksh93/ksh/issues/128

src/cmd/ksh93/sh/io.c:
- Add check for shp->subshare to the OpenSUSE patch.

src/cmd/ksh93/tests/io.sh:
- Add test.
2021-02-13 16:01:42 +00:00
Martijn Dekker
6f6b22016a command -x: tweak args list size detection (re: 9ddb45b1)
src/cmd/ksh93/features/externs: ARG_EXTRA_BYTES detection:
- Improve detection of extra bytes per argument: on every loop
  iteration, recalculate the size of the environment while taking
  the amount extra bytes we're currently trying into account. Also
  count arguments (argv[]) as they are stored in the same buffer.
  On 64-bit Linux with glibc, this now detects 9 extra bytes per
  argument instead of 8. An odd number (literally and figuratively)
  but apparently it needs it; I do think my method is correct now.
  On 64-bit Solaris and macOS, this still detects 8 extra bytes.
  (On 64-bit Linux with musl C library, it detects 0 bytes. Nice.)

src/cmd/ksh93/sh/path.c: path_xargs():
- Remove the kludge subtracting twice the size of the environment.
  With the feature test fixed, this should no longer fail on Linux.
- Take into account the size of the final null element in the
  argument and environment lists.

src/cmd/ksh93/tests/path.sh:
- Do not use awk for the test due to breakage in the system awks
  on Solaris/Illumos (hangs) and AIX & UnixWare (drops arguments).
  Instead, use (wait for it...) ksh. It's a bit slower, but works.
2021-02-13 00:08:33 +00:00
Martijn Dekker
41ebb55a3a Fix most of job control (-m/-o monitor) in scripts
If I haven't missed anything, this should make the non-interactive
aspects of job control in scripts work as expected, except for the
"<command unknown>" issue in the output of 'bg', 'fg' and 'jobs'
(which is not such a high priority as those commands are really
designed for interactive use).

Plus, I believe I now finally understand what these three are for:
* The job.jobcontrol variable is set to nonzero by job_init() in
  jobs.c if, and only if, the shell is interactive *and* managed to
  get control of the terminal. Therefore, any changing of terminal
  settings (tcsetpgrp(3), tty_set()) should only be done if
  job.jobcontrol is nonzero. This commit changes several checks for
  sh_isoption(SH_INTERACTIVE) to checks for job.jobcontrol for
  better consistency with this.
* The state flag, sh_isstate(SH_MONITOR), determines whether the
  bits of job control that are relevant for both scripts and
  interactive shells are active, which is mostly making sure that a
  background job gets its own process group (setpgid(3)).
* The shell option, sh_isoption(SH_MONITOR), is just that. When the
  user turns it on or off, the state flag is synched with it. It
  should usually not be directly checked for, as the state may be
  temporarily turned off without turning off the option.

Prior discussion:
https://www.mail-archive.com/austin-group-l@opengroup.org/msg06456.html

src/cmd/ksh93/bltins/typeset.c, src/cmd/ksh93/sh/args.c:
- Move synching the SH_MONITOR state flag with the SH_MONITOR
  shell option from b_set() (the 'set' builtin) to sh_applyopts()
  which is indirectly called from b_set() and is also used when
  parsing the shell invocation command line. This ensures -m is
  properly enabled in both scenarios.

src/cmd/ksh93/sh/jobs.c:
- job_init(): Do not refuse to initialise job control on
  non-interactive shells. Instead, skip everything that should only
  be done on interactive shells (i.e., everything to do with the
  terminal). This function is now even more of a mess than it was
  before, so refactoring may be desirabe at some point.
- job_close(), job_set(), job_reset(), job_wait(): Do not reset the
  terminal process group (tcsetpgrp()) if job.jobcontrol isn't on.

src/cmd/ksh93/sh/xec.c:
- sh_exec(): TFORK: For SIGINT handling, check the SH_MONITOR
  state flag, not the shell option.
- sh_exec(): TFORK: Do not turn off the SH_MONITOR state flag in
  forked children. The non-interactive part of job control should
  stay active. Instead, turn off the SH_INTERACTIVE state flag so
  we don't get interactive shell behaviour (i.e. job control noise
  on the terminal) in forked subshells.
- _sh_fork(), sh_ntfork(): Do not reset the terminal process group
  (tcsetpgrp()) if job.jobcontrol isn't on. Do not turn off the
  SH_MONITOR state flag in forked children.

src/cmd/ksh93/sh/subshell.c: sh_subfork():
- Do not turn off the monitor option and state in forked subshells.
  The non-interactive part of job control should stay active.

src/cmd/ksh93/bltins/misc.c: b_bg():
- Check isstate(SH_MONITOR) instead of sh_isoption(SH_MONITOR) &&
  job.jobcontrol before throwing a 'no job control' error.
  This fixes a minor bug: fg, bg and disown could quietly fail.

src/cmd/ksh93/tests/jobs.sh:
- Add tests for 'fg' with job control IDs (%%, %1) in scripts.
- Add test checking that a background job launched from a subsell
  with job control enabled correctly becomes the leader of its own
  process group.

Makes progress on: https://github.com/ksh93/ksh/issues/119
2021-02-12 06:51:27 +00:00
Martijn Dekker
37a18bab71 Fix ${ comsub; } killing job control
Another longstanding whopper of a bug in basic ksh93 functionality:
run a ${ shared-state; } command substitution twice and job control
promptly loses track of all your running jobs. New jobs are tracked
again until you run another two shared-state command substitutions.
This is in at least 93t+, 93u-, 93u+, 93v- and ksh2020.

$ sleep 300 &
[1]	56883
$ jobs						# OK
[1] +  Running                 sleep 300 &
$ v=${ echo hi1; }
$ jobs						# OK
[1] +  Running                 sleep 300 &
$ v=${ echo hi2; }
$ jobs						# Nothing!
$ fg
ksh: fg: no such job

src/cmd/ksh93/sh/subshell.c: sh_subshell():
- The current environment number shp->curenv (a.k.a. sh.curenv) was
  not being restored if the virtual subshell we're leaving is of
  the shared-state command substitution variety as it was wrongly
  considered to be part of the environment that didn't need
  restoring. This caused it to be out of sync with shp->jobenv
  (a.k.a. sh.jobenv) which did get restored from savedcurenv.
  Restore both from savedcurenv at the same time for any subshell.
  (How these numbers are used exactly remains to be discovered.)

src/cmd/ksh93/tests/jobs.sh:
- Added, with a test for this bug to start it off. There is no
  other test script where job control fits, and a lot more related
  fixes are anticipated: https://github.com/ksh93/ksh/issues/119
2021-02-11 13:41:40 +00:00
Martijn Dekker
2996d7ae7c Fix typos in <customtypecommand> --man self-documentation
src/cmd/ksh93/bltins/enum.c:
- enum_type[]: Fix typos; minor edit for style.
- enum_type[], enuminfo(): Make the list of supported values
  comma-separated, instead of using a comma at the start of each.

src/cmd/ksh93/sh/nvtype.c:
- sh_opttype[]: Fix typos.
2021-02-10 16:25:11 +00:00
Martijn Dekker
76ea18dcbd Fix disabling SHOPT_FIXEDARRAY (re: 2182ecfa)
It was easier than expected to fix this one. The many regression
test failures caused by disabling it were all due to one bug:
'typeset -p' output broke when building without this option.

src/cmd/ksh93/sh/nvtree.c: nv_attribute():
- In this function to print the attributes of a name-value pair,
  move four lines of code out of #if SHOPT_FIXEDARRAY...#endif that
  were inadvertently moved into the #if block in ksh93 2012-05-18.
  See the changes to nvtree.c in this multishell repo commit:
  aabab56a

src/cmd/ksh93/data/builtins.c:
- Update/rewrite 'typeset -a' documentation.
- Make it adapt to SHOPT_FIXEDARRAY.
- Fix a few typos.

src/cmd/ksh93/tests/arrays2.sh:
- Only one regression test needs a SHOPT_FIXEDARRAY check.

.github/workflows/ci.yml:
- Disable SHOPT_FIXEDARRAY when regression-testing without SHOPTs.
- Enable xtrace, add ':' commands for traced comments. This should
  make the CI runner output logs a little more readable.
2021-02-10 04:48:56 +00:00
Martijn Dekker
2182ecfa08 Fix compile/regress fails on compiling without SHOPT_* options
Many compile-time options were broken so that they could not be
turned off without causing compile errors and/or regression test
failures. This commit now allows the following to be disabled:

SHOPT_2DMATCH    # two dimensional ${.sh.match} for ${var//pat/str}
SHOPT_BGX        # one SIGCHLD trap per completed job
SHOPT_BRACEPAT   # C-shell {...,...} expansions (, required)
SHOPT_ESH        # emacs/gmacs edit mode
SHOPT_HISTEXPAND # csh-style history file expansions
SHOPT_MULTIBYTE  # multibyte character handling
SHOPT_NAMESPACE  # allow namespaces
SHOPT_STATS      # add .sh.stats variable
SHOPT_VSH        # vi edit mode

The following still break ksh when disabled:

SHOPT_FIXEDARRAY # fixed dimension indexed array
SHOPT_RAWONLY    # make viraw the only vi mode
SHOPT_TYPEDEF    # enable typeset type definitions

Compiling without SHOPT_RAWONLY just gives four regression test
failures in pty.sh, but turning off SHOPT_FIXEDARRAY and
SHOPT_TYPEDEF causes compilation to fail. I've managed to tweak the
code to make it compile without those two options, but then dozens
of regression test failures occur, often in things nothing directly
to do with those options. It looks like the separation between the
code for these options and the rest was never properly maintained.
Making it possible to disable SHOPT_FIXEDARRAY and SHOPT_TYPEDEF
may involve major refactoring and testing and may not be worth it.

This commit has far too many tweaks to list. Notables fixes are:

src/cmd/ksh93/data/builtins.c,
src/cmd/ksh93/data/options.c:
- Do not compile in the shell options and documentation for
  disabled features (braceexpand, emacs/gmacs, vi/viraw), so the
  shell is not left with no-op options and inaccurate self-doc.

src/cmd/ksh93/data/lexstates.c:
- Comment the state tables to associte them with their IDs.
- In the ST_MACRO table (sh_lexstate9[]), do not make the S_BRACE
  state for position 123 (ASCII for '{') conditional upon
  SHOPT_BRACEPAT (brace expansion), otherwise disabling this causes
  glob patterns of the form {3}(x) (matching 3 x'es) to stop
  working as well -- and that is ksh globbing, not brace expansion.

src/cmd/ksh93/edit/edit.c: ed_read():
- Fixed a bug: SIGWINCH was not handled by the gmacs edit mode.

src/cmd/ksh93/sh/name.c: nv_putval():
- The -L/-R left/right adjustment options to typeset do not count
  zero-width characters. This is the behaviour with SHOPT_MULTIBYTE
  enabled, regardless of locale. Of course, what a zero-width
  character is depends on the locale, but control characters are
  always considered zero-width. So, to avoid a regression, add some
  fallback code for non-SHOPT_MULTIBYTE builds that skips ASCII
  control characters (as per iscntrl(3)) so they are still
  considered to have zero width.

src/cmd/ksh93/tests/shtests:
- Export the SHOPT_* macros from SHOPT.sh to the tests as
  environment variables, so the tests can check for them and decide
  whether or how to run tests based on the compile-time options
  that the tested binary was presumably compiled with.
- Do not run the C.UTF-8 tests if SHOPT_MULTIBYTE is not enabled.

src/cmd/ksh93/tests/*.sh:
- Add a bunch of checks for SHOPT_* env vars. Since most should
  have a value 0 (off) or 1 (on), the form ((SHOPT_FOO)) is a
  convenient way to use them as arithmetic booleans.

.github/workflows/ci.yml:
- Make GitHub do more testing: run two locale tests (Dutch and
  Japanese UTF-8 locales), then disable all the SHOPTs that we can
  currently disable, recompile ksh, and run the tests again.
2021-02-08 22:02:45 +00:00
Martijn Dekker
403e864ac8 Disallow >;word and <>;word for 'redirect' (re: 7b59fb80, 7b82c338)
The >;word and <>;word redirection operators cannot be used with
the 'exec' builtin, but the 'redirect' builtin (which used to be
an alias of 'command exec') permitted them. However, they do not
have the documented effect of the added ';'. So this commit blocks
those operators for 'redirect' as they are blocked for 'exec'.

It also tweaks redirect's error message if a non-redirection
argument is encountered.

src/cmd/ksh93/sh/parse.c: simple():
- Set the lexp->inexec flag for SYSREDIR (redirect) as well as
  SYSEXEC (exec). This flag is checked for in sh_lex() (lex.c) to
  throw a syntax error if one of these two operators is used.

src/cmd/ksh93/sh.1, src/cmd/ksh93/data/builtins.c:
- Documentation tweaks.

src/cmd/ksh93/sh/xec.c, src/cmd/ksh93/bltins/misc.c:
- When 'redirect' gives an 'incorrect syntax' (e_badsyntax) error
  message, include the first word that was found not to be a valid
  redirection. This is simply the first argument, as redirections
  are removed from the arguments list.

src/cmd/ksh93/tests/io.sh:
- Update test to reflect new error message format.
2021-02-07 03:23:56 +00:00
Martijn Dekker
7b59fb805c Fix 'redirect {var}>file' in subshell (re: 7b82c338)
Permanent redirections of that form broke in subshells when used
with the 'redirect' command, because I had overlooked one instance
where the new 'redirect' builtin needs to match the behaviour of
the 'exec' builtin.

src/cmd/ksh93/tests/io.sh: sh_exec():
- Do not restore file descriptors in (virtual) subshells for
  'redirect' just as this isn't done for 'exec'.

src/cmd/ksh93/tests/io.sh:
- Add regression test for this bug.
- Complete the test for f9427909 which I committed prematurely.

Fixes: https://github.com/ksh93/ksh/issues/167
2021-02-05 05:38:38 +00:00
Martijn Dekker
f9427909dc Make redirections like {varname}>file work with brace expansion off
This is some nonsense: redirections that store a file descriptor
greater than 9 in a variable, like {var}<&2 and the like, stopped
working if brace expansion was turned off. '{var}' is not a brace
expansion as it doesn't contain ',' or '..'; something like 'echo
{var}' is always output unexpanded. And redirections and brace
expansion are two completely unrelated things. It wasn't documented
that these redirections require the -B/braceexpand option, either.

src/cmd/ksh93/sh/lex.c: sh_lex():
- Remove incorrect check for braceexpand option before processing
  redirections of this form.

src/cmd/ksh93/COMPATIBILITY:
- Insert a brief item mentioning this.

src/cmd/ksh93/sh.1:
- Correction: these redirections do not yield a file descriptor >
  10, but > 9, a.k.a. >= 10.
- Add a brief example showing how these redirections can be used.

src/cmd/ksh93/tests/io.sh:
- Add a quick regression test.
2021-02-05 05:08:39 +00:00
Martijn Dekker
c709868572 process substitution: improve fifo error handling (re: ab5dedde)
src/cmd/ksh93/sh/args.c: sh_argprocsub():
- Fix compiler warnings with SHOPT_DEVFD on by including "io.h".
- Without SHOPT_DEVFD, the FIFO code didn't consider that libast's
  pathtemp(3) may also fail and return null. Add a check for this.
2021-02-04 20:47:45 +00:00
Martijn Dekker
a410bc480f autoload: Add loop detection
It was trivial to crash ksh by making an autoloaded function
definition file autoload itself, causing a stack overflow due to
infinite recursion. This commit adds loop detection that stops a
function that is being autoloaded from autoloading itself either
directly or indirectly, without removing the ability of autoloaded
function definition files to autoload other functions.

src/cmd/ksh93/sh/path.c: funload():
- Detect loops by checking if the path of a function to be
  autoloaded was already added to a new internal static tree,
  and if not, adding it while the function is being loaded.

src/cmd/ksh93/tests/path.sh:
- Add regression test.
- Tweak a couple of others to be freeze- and crash-proof.

NEWS:
- Add this fix + a forgotten entry for the previous fix (6f3b23e6).

Fixes: https://github.com/ksh93/ksh/issues/136
2021-02-04 16:28:19 +00:00
Martijn Dekker
6f3b23e6f4 Fix crash on trying a very long nonexistent command
Reproducer from @Saikiran-m:
| ~# sh -c `perl -e 'print "a"x100000'`
|  genunix: NOTICE: core_log: sh[1221] core dumped: /var/cores/core.sh.0.1602153496
| Memory fault(coredump)

The crash was in trying to decide whether the name was suitable for
autoloading as a function on $FPATH. This calls strmatch() to check
the name against a regex for valid function name. But the libast
regex code is not designed optimally and uses too much recursion,
limiting the length of the strings it's able to cope with.

src/cmd/ksh93/sh/path.c: path_search():
- Before calling strmatch(), check that the name is shorter than
  256 bytes. The maximum length of file names on Linux and macOS is
  255 bytes, so an autoload function can't have a name longer than
  that anyway.

src/cmd/ksh93/tests/path.sh:
- Add test for this bug.
- Tweak 'command -x' test to not leave a hanging process on Ctrl+C.

Fixes: https://github.com/ksh93/ksh/issues/144
2021-02-04 05:03:40 +00:00
Martijn Dekker
9ddb45b12d command -x: fix 'args list too long' on Linux, again (re: 8f5235a5)
This time it was failing on a 64-bit Debian Linux system with very
few and short environment variables. Sigh.

src/cmd/ksh93/sh/path.c:
- Combine the strategy from 63979488 with that of 8f5235a5.
2021-02-03 20:07:47 +00:00
Martijn Dekker
8f5235a53f fix 'args list too long' on Linux, again (re: 63979488)
That fix turned out to be insufficient as NixOS has huge
environment variable lists because (due to each software package
being installed in its own directory tree) it has to keep dozens
of directories in variables like XDG_CONFIG_DIRS and others.
The 'command -x' regression test was failing on NixOS.

src/cmd/ksh93/sh/path.c:
- Different strategy. Leave twice the size of the existing
  environment free.
2021-02-03 19:22:35 +00:00
Martijn Dekker
52067c3d37 Backport atomic job locking from ksh 93v- beta
Something similar was previously done in 07cc71b8 from a Debian
patch, and eventually reverted; it redefined the ast atomic
functions asoincint() and asodecint() to be gcc-specific. This
imports the upstream version from the ksh 93v- beta instead.

This commit is based on an OpenSUSE patch:
https://build.opensuse.org/package/view_file/shells/ksh/ksh93-joblock.dif

src/cmd/ksh93/include/jobs.h:
- Replace job locking mechanism with the 93v- version which uses
  the atomic libast functions asoincint(), asogetint() and
  asodecint(). See: src/lib/libast/man/aso.3

src/cmd/ksh93/sh/jobs.c: job_subsave():
- Revert gcc optimiser bug workaround from c258a04f.
  It should now be unnecessary.
2021-02-02 17:38:19 +00:00
Martijn Dekker
63979488e6 command -x: fix 'arg list too long' on Linux (re: 66e1d446)
I got one intermittent regression test failure due to 'argument
list too long' on a Debian x86_64 system.

src/cmd/ksh93/sh/path.c: path_xargs():
- Leave extra argument space for systems that need extra bytes:
  1KiB per extra byte, with a minimum of 2KiB (the old value).
2021-02-02 15:23:21 +00:00
Martijn Dekker
4645e6ad33 job_reset(): fix for resetting foreground process group
Patch from OpenSUSE:
https://build.opensuse.org/package/view_file/shells/ksh/ksh93-foreground-prgrp.dif

src/cmd/ksh93/sh/jobs.c: job_reset():
- Only reset the foreground process group associated with the
  current job if the current job's process ID is different from the
  terminal's foreground process group ID.
2021-02-02 12:49:00 +00:00
Martijn Dekker
9f980ba65a sh_setmatch(): fix node size calculation
This fixes the function that sets ${.sh.match}. Patch from OpenSUSE:
https://build.opensuse.org/package/view_file/shells/ksh/ksh93-limit-name-len.dif

src/cmd/ksh93/sh/init.c: sh_setmatch():
- Fix node size calculation, possibly preventing data corruption.

src/cmd/ksh93/include/ulimit.h: Limit_t:
- Defining the 'name' struct member as 'char name[16]' makes
  no sense as the name is being initialised statically in
  data/limits.c; just make it a 'char *name' pointer.
2021-02-02 11:52:54 +00:00
Martijn Dekker
fa3a3d7955 sh_ntfork(): handle SIGTSTP along with SIGTTIN and SIGTTOU
This applies a patch from OpenSUSE that makes SIGTSTP handling
consistent with that of SIGTTIN and SIGTTOU.
https://build.opensuse.org/package/view_file/shells/ksh/ksh93-signals.dif
2021-02-02 04:25:46 +00:00
Martijn Dekker
0e4c4d619d Fix minor typeset attribute regressions (re: 95fe07d8, fdb9781e)
This fixes the following regressions marked TODO in attributes.sh:

$ typeset -L 13 bar; readonly bar; typeset -p bar
typeset -r -L 0 foo		# exp.: typeset -r -L 13 foo
$ typeset -R 13 bar; readonly bar; typeset -p bar
typeset -r -R 0 bar		# exp.: typeset -r -R 13 bar
$ typeset -Z 13 baz; readonly baz; typeset -p baz
typeset -r -Z 0 -R 0 baz	# exp.: typeset -r Z 13 -R 13 baz

I've discovered that these were briefly fixed between fdb9781e (Red
Hat patch for typeset -xu/-xl) and 95fe07d8 (reversal of patch,
different -xu/-xl fix, but reintroduced these regressions).

src/cmd/ksh93/sh/name.c: nv_newattr():
- Replace check from 95fe07d8 with a new one that combines its
  approach with that of fdb9781e: do not change size (and hence
  return early) if NV_RDONLY and/or NV_EXPORT are the only
  attributes that are changing.

src/cmd/ksh93/tests/attributes.sh:
- Enable the TODO regression tests.
2021-02-02 02:20:37 +00:00
Martijn Dekker
5491fe9724 Correctly block invalid values for arrays of an enum type
This fixes part of https://github.com/ksh93/ksh/issues/87:

Scalar arrays (-a) and associative arrays (-A) of a type created by
'enum' did not consistently block values not specified by the enum
type, yielding corrupted results.

An expansion of type "${array[@]}" yielded random numbers instead
of values for associative arrays of a type created by 'enum'.

This does not yet fix another problem: ${array[@]} does not yield
all values for associative enum arrays.

src/cmd/ksh93/bltins/enum.c: put_enum():
- Always throw an error if the value is not in the list of possible
  values for an enum type. Remove incorrect check for the NV_NOFREE
  flag. Whatever that was meant to accomplish, I've no idea.

src/cmd/ksh93/sh/array.c: nv_arraysettype():
- Instead of sh_eval()ing a shell assignment, use nv_putval()
  directly. Also use the stack (see src/lib/libast/man/stk.3)
  instead of malloc to save the value; it's faster and will be
  auto-freed at some point. This shortens the function and makes it
  faster by not entering into a whole new shell context -- which
  also fixes another problem: the error message from put_enum()
  didn't cause the shell to exit for indexed enum arrays.

src/cmd/ksh93/sh/name.c: nv_setlist():
- Apply a patch from David Korn that correctly sets the data type
  for associative arrays, fixing the ${array[@]} expansion yielding
  random numbers. Thanks to @JohnoKing for the pointer.
  https://github.com/ksh93/ksh/issues/87#issuecomment-662613887
  https://www.mail-archive.com/ast-developers@lists.research.att.com/msg00697.html

src/cmd/ksh93/tests/enum.sh:
- Add tests checking that invalid values are correctly blocked for
  indexed and associative arrays of an enum type.

Makes progress on: https://github.com/ksh93/ksh/issues/87
2021-02-01 16:57:43 +00:00
Martijn Dekker
6a0e9a1a75 Tweak and regress-test 'command -x' (re: 66e1d446)
Turns out the assumption I was operating on, that Linux and macOS
align arguments on 32 or 64 bit boundaries, is incorrect -- they
just need some extra bytes per argument. So we can use a bit more
of the arguments buffer on these systems than I thought.

src/cmd/ksh93/features/externs:
- Change the feature test to simply detect the # of extra bytes per
  argument needed. On *BSD and commercial Unices, ARG_EXTRA_BYTES
  shows as zero; on Linux and macOS (64-bit), this yields 8. On
  Linux (32-bit), this yields 4.

src/cmd/ksh93/sh/path.c: path_xargs():
- Do not try to calculate alignment, just add ARG_EXTRA_BYTES to
  each argument.
- Also add this when substracting the length of environment
  variables and leading and trailing static command arguments.

src/cmd/ksh93/tests/path.sh:
- Test command -v/-V with -x.
- Add a robust regression test for command -x.

src/cmd/ksh93/data/builtins.c, src/cmd/ksh93/sh.1:
- Tweak docs. Glob patterns also expand to multiple words.
2021-02-01 02:19:02 +00:00
Martijn Dekker
ede479967f resolve/remove USAGE_LICENSE macros; remove repetitive (c) strings
This takes another small step towards disentangling the build
system from the old AT&T environment. The USAGE_LICENSE macros with
author and copyright information, which was formerly generated
dynamically for each file from a database, are eliminated and the
copyright/author information is instead inserted into the AST
getopt usage strings directly.

Repetitive license/copyright information is also removed from the
getopt strings in the builtin commands (src/lib/libcmd/*.c and
src/cmd/ksh93/data/builtins.c). There's no need to include 55
identical license/copyright strings in the ksh binary; one (in the
main ksh getopt string, shown by ksh --man) ought to be enough!
This makes the ksh binary about 10k smaller.

It does mean that something like 'enum --author', 'typeset
--license' or 'shift --copyright' will now not show those notices
for those builtins, but I doubt anyone will care.
2021-01-31 11:00:49 +00:00
Martijn Dekker
66e1d44642 command -x: fix efficiency; always run external cmd (re: acf84e96)
This commit fixes 'command -x' to adapt to OS limitations with
regards to data alignment in the arguments list. A feature test is
added that detects if the OS aligns the argument on 32-bit or
64-bit boundaries or not at all, allowing 'command -x' to avoid
E2BIG errors while maximising efficiency.

Also, as of now, 'command -x' is a way to bypass built-ins and
run/query an external command. Built-ins do not limit the length of
their argument list, so '-x' never made sense to use for them. And
because '-x' hangs on Linux and macOS on every ksh93 release
version to date (see acf84e96), few use it, so there is little
reason not to make this change.

Finally, this fixes a longstanding bug that caused the minimum exit
status of 'command -x' to be 1 if a command with many arguments was
divided into several command invocations. This is done by replacing
broken flaggery with a new SH_XARG state flag bit.

src/cmd/ksh93/features/externs:
- Add new C feature test detecting byte alignment in args list.
  The test writes a #define ARG_ALIGN_BYTES with the amount of
  bytes the OS aligns arguments to, or zero for no alignment.

src/cmd/ksh93/include/defs.h:
- Add new SH_XARG state bit indicating 'command -x' is active.

src/cmd/ksh93/sh/path.c: path_xargs():
- Leave extra 2k in the args buffer instead of 1k, just to be sure;
  some commands add large environment variables these days.
- Fix a bug in subtracting the length of existing arguments and
  environment variables. 'size -= strlen(cp)-1;' subtracts one less
  than the size of cp, which makes no sense; what is necessary is
  to substract the length plus one to account for the terminating
  zero byte, i.e.: 'size -= strlen(cp)+1'.
- Use the ARG_ALIGN_BYTES feature test result to match the OS's
  data alignment requirements.
- path_spawn(): E2BIG: Change to checking SH_XARG state bit.

src/cmd/ksh93/bltins/whence.c: b_command():
- Allow combining -x with -p, -v and -V with the expected results
  by setting P_FLAG to act like 'whence -p'. E.g., as of now,
	command -xv printf
  is equivalent to
	whence -p printf
  but note that 'whence' has no equivalent of 'command -pvx printf'
  which searches $(getconf PATH) for a command.
- When -x will run a command, now set the new SH_XARG state flag.

src/cmd/ksh93/sh/xec.c: sh_exec():
- Change to using the new SH_XARG state bit.
- Skip the check for built-ins if SH_XARG is active, so that
  'command -x' now always runs an external command.

src/lib/libcmd/date.c, src/lib/libcmd/uname.c:
- These path-bound builtins sometimes need to run the external
  system command by the same name, but they did that by hardcoding
  an unportable direct path. Now that 'command -x' runs an external
  command, change this to using 'command -px' to guarantee using
  the known-good external system utility in the default PATH.
- In date.c, fix the format string passed to 'command -px date'
  when setting the date; it was only compatible with BSD systems.
  Use the POSIX variant on non-BSD systems.
2021-01-30 06:53:19 +00:00
Martijn Dekker
674a0c3559 Another lexical fix for here-documents (re: 6e515f1d)
OpenSUSE patch from:
https://build.opensuse.org/package/view_file/shells/ksh/ksh93-heredoc.dif

src/cmd/ksh93/sh/lex.c: here_copy():
- Do not potentially seek back a zero or negative length.
2021-01-28 06:32:42 +00:00
Martijn Dekker
4604df9ada Stack robustness fixes from OpenSUSE
Three OpenSUSE patches from:
https://build.opensuse.org/package/show/shells/ksh

As usual, the relevant bug is not currently public:
https://bugzilla.opensuse.org/show_bug.cgi?id=844071

src/cmd/ksh93/sh/xec.c: sh_debug()/sh_exec():
- Fix stk restoration. [bnc#844071]

src/lib/libast/misc/stk.c:
- Fix stk aliasing code. [bnc#844071]
  (ksh93-stkalias.dif)
- Make a unknown location fatal in stkset() so that we get a core
  dump right away instead of later in an unrelated part of code.
  (ksh93-stkset-abort.dif)

src/lib/libast/man/stk.3,
src/lib/libast/man/stak.3:
- Update manual with new stkset() behaviour. (93u+m addition)
  (Note that stak is implemented as macros that translate to stk)
2021-01-28 06:18:44 +00:00
Martijn Dekker
288b6c6517 Fix various possible uses of uninitialised variables
Patch from OpenSUSE, slightly adapted for 93u+m. Source:
https://build.opensuse.org/package/view_file/shells/ksh/ksh93-uninitialized.dif
2021-01-28 04:54:41 +00:00
Martijn Dekker
c52cb93999 sh_funscope(): Fix possible dereference of null pointer
Patch from OpenSUSE:
https://build.opensuse.org/package/view_file/shells/ksh/ksh93-unset-f.dif
2021-01-28 04:38:48 +00:00
Martijn Dekker
3ad4307054 name.c: fix possible crash in attstore()
From OpenSUSE:
https://build.opensuse.org/package/view_file/shells/ksh/ksh93-env.dif
(the init.c fix there is already done differently in 3654ee73)

src/cmd/ksh93/sh/name.c: attstore():
- Check nv_mapchar() returns a non-null pointer before using it.
2021-01-28 04:01:49 +00:00