Reproducer (symptoms on at least macOS and FreeBSD):
$ mkfifo f
$ echo foo > f
(press Ctrl+Z)
^Zksh: f: cannot create [Interrupted system call]
Abort
The shell either aborts (dev builds) or crashes with 'Illegal
instruction' (release builds). This is consistent with
UNREACHABLE() being reached.
Backtrace:
0 libsystem_kernel.dylib __kill + 10
1 ksh sh_done + 836 (fault.c:678)
2 ksh sh_fault + 1324
3 libsystem_platform.dylib _sigtramp + 29
4 dyld ImageLoaderMachOCompressed::resolve(ImageLoader::LinkContext const&, char const*, unsigned ch
5 libsystem_c.dylib abort + 127
6 ksh sh_redirect + 3576 (io.c:1356)
7 ksh sh_exec + 7231 (xec.c:1308)
8 ksh exfile + 3247 (main.c:607)
9 ksh sh_main + 3551 (main.c:368)
10 ksh main + 38 (pmain.c:45)
11 libdyld.dylib start + 1
This means that UNREACHABLE() is actually reached here:
ksh/src/cmd/ksh93/sh/io.c
1351: if((fd=sh_open(tname?tname:fname,o_mode,RW_ALL)) <0)
1352: {
1353: errormsg(SH_DICT,ERROR_system(1),((o_mode&O_CREAT)?e_create:e_open),fname);
1354: UNREACHABLE();
1355: }
The cause is that, in the following section of code in sh_fault():
ksh/src/cmd/ksh93/sh/fault.c
183: #ifdef SIGTSTP
184: if(sig==SIGTSTP)
185: {
186: sh.trapnote |= SH_SIGTSTP;
187: if(pp->mode==SH_JMPCMD && sh_isstate(SH_STOPOK))
188: {
189: sigrelease(sig);
190: sh_exit(SH_EXITSIG);
191: return;
192: }
193: }
194: #endif /* SIGTSTP */
...sh_exit() is not getting called and the function will not return
because the SH_STOPOK bit is not set while the shell is blocked
waiting to write to a FIFO.
Even if sh_exit() did get called, that would not fix it, because
that function also checks for the SH_STOPOK bit and returns without
doing a longjmp if the signal is SIGTSTP and the SH_STOPOK bit is
not set. That is direct the reason why UNREACHABLE() was raeched:
errormsg() does call sh_exit() but sh_exit() then does not longjmp.
src/cmd/ksh93/sh/fault.c: sh_fault():
- To avoid the crash, we simply need to return from sh_fault() if
SH_STOPOK is off, so that the code path does not continue, no
error message is given on Ctrl+Z, UNREACHABLE() is not reached,
and the shell resumes waiting on trying to write to the FIFO.
The sh.trapnote flag should not be set if we're not going to
process the signal. This makes ksh behave like all other shells.
Resolves: https://github.com/ksh93/ksh/issues/464
Reproducer:
$ ksh -c 'unset PWD; (cd /); :'
Memory fault
The shell crashes because b_cd() is testing the value of the PWD
variable without checking if there is one.
src/cmd/ksh93/sh/path.c: path_pwd():
- Never return an unfreeable pointer to e_dot; always return a
freeable pointer. This fixes another corner-case crashing bug.
- Make sure the PWD variable gets assigned a value if it doesn't
have one, even if it's the "." fallback. However, if the PWD is
inaccessible but we did inherit a $PWD value that starts with a
/, then use the existing $PWD value as this will help the shell
fail gracefully.
src/cmd/ksh93/bltins/cd_pwd.c:
- b_cd(): When checking if the PWD is valid, use the sh.pwd copy
instead of the PWD variable. This fixes the crash above.
- b_cd(): Since path_pwd() now always returns a freeable value,
free sh.pwd unconditionally before setting the new value.
- b_pwd(): Not only check that path_pwd() returns a value starting
with a slash, but also verify it with test_inode() and error out
if it's wrong. This makes the 'pwd' command useful for checking
that the PWD is currently accessible.
src/cmd/ksh93/data/msg.c:
- Change e_pwd error message for accuracy and clarity.
This backports two minor additions to the 'read' built-in from ksh
93v-: '-a' is now the same as '-A' and '-u p' is the same as '-p'.
This is for compatibility with some 93v- or ksh2020 scripts.
Note that their change to the '-p' option to support both prompts
and reading from the coprocess was *not* backported because we
found it to be broken and unfixable. Discussoin at:
https://github.com/ksh93/ksh/issues/463
src/cmd/ksh93/bltins/read.c: b_read():
- Backport as described above.
- Rename the misleadingly named 'name' variable to 'prompt'.
It points to the prompt string, not to a variable name.
src/cmd/ksh93/data/builtins.c: sh_optpwd[]:
- Add -a as an alterative to -A. All that is needed is adding '|a'
and optget(3) will automatically convert it to 'A'.
- Change -u from a '#' (numeric) to ':' option to support 'p'. Note
that b_read() now needs a corresponding strtol() to convert file
descriptor strings to numbers where applicable.
- Tweaks.
src/cmd/ksh93/sh.1:
- Update accordingly.
- Tidy up the unreadable mess that was the 'read' documentation.
The options are now shown in a list.
From README:
FILESCAN on Experimental option that allows fast reading of files
using while < file;do ...; done and allowing fields in
each line to be accessed as positional parameters.
As SHOPT_FILESCAN has been enabled by default since ksh 93l
2001-06-01, the filescan loop is now documented in the manual page
and the compile-time option is no longer considered experimental.
We must disable this at runtime if --posix is active because it
breaks a portable use case: POSIXly, 'while <file; do stuff; done'
repeatedly excutes 'stuff' while 'file' can successfully be opened
for reading, without actually reading from 'file'.
This also backports a bugfix from the 93v- beta. Reproducer:
$ echo 'one two three' >foo
$ while <foo; do printf '[%s] ' "$@"; echo; done
[one two three]
Expected output:
[one] [two] [three]
The bug is that "$@" acts like "$*", joining all the positional
parameters into one word though it should be generating one word
for each.
src/cmd/ksh93/sh/macro.c: varsub():
- Backport fix for the bug described above. I do not understand the
opaque macro.c code well enough yet to usefully describe the fix.
src/cmd/ksh93/sh/xec.c: sh_exec():
- Improved sanity check for filescan loop: do not recognise it if
the simple command includes variable assignments, more than one
redirection, or an output or append redirection.
- Disable filescan loops if --posix is active.
- Another 93v- fix: handle interrupts (errno==EINTR) when closing
the input file.
As so often with SHOPT_* compile-time options, not all the relevant
code was actually conditional on the option macro. After removing
SHOPT_ENV, the arguments to sh_putenv() and env_delete() macro
calls are dead code and, after removing that, the sh.env variable
is unused.
src/cmd/ksh93/{sh/path.c,include/shell.h}:
- The sh.defpathlist variable is never set once, which makes every
use of this variable unnecessary (as it's always null). This
commit removes sh.defpathlist while also fixing a possible memory
leak (there was another location where defpathinit() was invoked
without saving the returned pointer, causing a memory leak).
In UTF-8 locales, ksh breaks when a KEYBD trap is active, even a
dummy no-op one like 'trap : KEYBD'. Entering multi-byte characters
fails (the input is interrupted and a new prompt is displayed) and
pasting content with multi-byte characters produces corrupted
results.
The cause is that the KEYBD trap code is not multibyte-ready.
Unfortunately nobody yet understands the edit.c code well enough
to implement a proper fix. Pending that, this commit implements
a workaround that at least avoids breaking the shell.
src/cmd/ksh93/edit/edit.c: ed_getchar():
- When a multi-byte locale is active, do not trigger the the KEYBD
trap except for ASCII characters (1-127).
Resolves: https://github.com/ksh93/ksh/issues/307
macOS 12.2.1 doesn't seem to like the -M, -v or -d ulimit options:
sh_match.sh[502]: FAIL: test_xmlfragment1/0/testfile1.xml:
Expected empty stderr, got $'test1_script.sh[2]: ulimit: 1048576:
limit exceeded [Invalid argument]\ntest1_script.sh[3]: ulimit:
1048576: limit exceeded [Invalid argument]\ntest1_script.sh[4]:
ulimit: 1048576: limit exceeded [Invalid argument]'
The 'Invalid argument' addition is caused by errno==EINVAL and
suggests the OS either doesn't support setting this limit, or
support for it was somehow disabled.
src/cmd/ksh93/tests/sh_match.sh:
- Redirect standard error for ulimit commands to 2>/dev/null. If
they fail it's pretty inconsequential and it's not related to
actual ${.sh.match} testing at all.
Resolves: https://github.com/ksh93/ksh/issues/459
Thanks to @posguy99 for the report.
Sfio may theoretically be compiled with threads support using a
separate AT&T library called Vthread, also by Kiem-Phong Vo. That
library was never shipped in the AST distribution, though. It is
only available with a standalone version of Sfio.
The only standalone Sfio version with Vthread that I've found is
from 2005, mirrored at <https://github.com/lichray/sfio>. More
recent versions never seem to have made it out of the defunct AT&T
software download site.
Even if they weren't, the rest of libast doesn't support threads,
and at this point it never will, so for our purposes the Sfio
threads code is never going to be usable. Meanwhile, macros such as
SFMTXENTER and SFMTXRETURN make the code a lot harder to read. And
not quite all threading code is disabled; some of it is dead code
that is getting compiled in.
Chances are that code now won't work properly in any case as we've
not had any chance to test it as we were making changes. Bit rot
has surely set in by now.
So this commit expands all the sfio/stdio threading-related macros
to their non-threads fallbacks (which is null for most of them, but
not all), deletes dead mutex-related code and struct fields, and
removes the related documentation from the sfio.3 man page. Unless
I did something wrong, there should be no change in behaviour.
The -f/--noglob shell option is documented simply as: "Disables
pathname expansion." But after 'set -f' on an interactive shell,
command completion and file name completion also stop working. This
is because they internally use the pathname expansion mechanism.
But it is not documented anywhere that 'set -f' disables
completion; it's just a side effect of an implementation detail.
Though ksh has always acted like this, I think it should change
because it's not useful or expected behaviour. Other shells like
bash, yash or zsh don't act like this.
src/cmd/ksh93/sh/expand.c,
src/cmd/ksh93/sh/macro.c:
- Allow the SH_COMPLETE (command completion) or SH_FCOMPLETE
(file name completion) state bit to override SH_NOGLOB in
path_generate() and in sh_macexpand().
It probably won't make a difference since the 'sleep' is run in the
background, but let's change 'sleep .5 &' back to the original
'sleep 1 &' from the 93u+ 2012-08-01 version and see what happens.
See: https://github.com/ksh93/ksh/issues/344#issuecomment-982219206
The >&- redirection subshell leak fixed in 6304dfce still existed
for shared-state ${ command substitutions; } a.k.a. subshares,
which cannot be forked.
I previously noticed that sh_subsavefd() saves the FD state even
for subshares. That seems logically incorrect as subshares share
their state with the invoking environment by definition.
src/cmd/ksh93/sh/subshell.c: sh_subsavefd():
- Sure enough, adding a check for !sh.subshare fixes the bug.
- Use the sh.subshell counter and not the subshell data pointer to
check for a virtual subshell. If the shell is reinitialised in a
fork to execute a new script (see 0af81992), any parent virtual
subshell data is currently not cleared as it is locally scoped to
subshell.c, so that check would be incorrect then.
src/cmd/ksh93/sh/io.c: sh_redirect:
- Remove now-redundant (and actually incorrectly placed) check for
sh.subshare added in fb755163.
Reproducer:
exec 9>&1
( { exec 9>&1; } 9>&- )
echo "test" >&9 # => 9: cannot open [Bad file descriptor]
The 9>&- incorrectly persists beyond the { } block that it
was attached to *and* beyond the ( ) subshell. This is yet another
bug with non-forking subshells; forking it with something like
'ulimit -t unlimited' works around the bug.
In over a year we have not been able to find a real fix, but I came
up with a workaround that forks a virtual subshell whenever it
executes a code block with a >&- or <&- redirection attached. That
use case is obscure enough that it should not cause any performance
regression except in very rare corner cases.
src/cmd/ksh93/sh/xec.c: sh_exec(): TSETIO:
- This is where redirections attached to code blocks are handled.
Check for a >&- or <&- redirection using bit flaggery from
shnodes.h and fork if we're executing such in a virtual subshell.
Resolves: https://github.com/ksh93/ksh/issues/161
Thanks to @ko1nksm for the bug report.
Notable changes:
- Remove SHOPT_PFSH compile-time option and associated code.
This was meant to work with Solaris rights profiles, see:
https://docs.oracle.com/cd/E23824_01/html/821-1461/profiles-1.html#REFMAN1profiles-1
But it has been obsolete for years as Solaris stopped using
it in its shipped ksh several OS versions ago, preferring a
library-based wrapper around ksh and other shells.
Nonetheless I experimented with the option on Solaris 11.4.
Result: no external command will run; output of unitialised
memory in error message. So it's already fallen victim to bit
rot. There's nothing interesting here, so just get rid.
- Remove SHOPT_TYPEDEF compile-time option (but keep the code!).
Turning it off caused the build to fail. It may be possible to
fix it, but the type definition code is integral to ksh now (e.g.
'enum' depends on much of it) so it makes no sense to disable it.
This was removed in the ksh 93v- beta version as well.
- Remove nv_close() calls and remove nv_close() documentation from
the nval.3 man page. This function is a dummy, present without
any changes since the beginning of the ast-open-archive repo in
1995. The comment was: "Currently this is a dummy, but someday
will be needed for reference counting". 27 or more years later,
it's time to admit it's never going to happen. (And of course,
nv_close() calls were not being used with anything resembling
consistency.)
- Add a null nv_close() macro to nval.h for compatibility with
third party code that follows the old documentation.
- Add a few missing regression tests.
This small commit replaces one instance of memcmp with strncmp to
fix one of the buffer overflows that causes 'typeset -p .sh.type'
to crash (see also https://github.com/ksh93/ksh/issues/456).
This commit backports all of the relevant .sh.match bugfixes from
ksh93v-. Most of the .sh.match rewrite is from versions 2012-08-24
and 2012-10-04, with patches from later releases of 93v- and
ksh2020 also applied. Note that there are still some remaining bugs
in .sh.match, although now the total count of .sh.match bugs should
be less that before.
These are the relevant changes in the ksh93v- changelog that were
backported:
12-08-07 .sh.match no longer gets set for patterns in PS4 during
set -x.
12-08-10 Rewrote .sh.match expansions fixing several bugs and
improving performance.
12-08-22 .sh.match now handles subpatterns that had no matches with
${var//pattern} correctly.
12-08-21 A bug in setting .sh.match after ${var//pattern/string}
when string is empty has been fixed.
12-08-21 A bug in setting .sh.match after [[ string == pattern ]]
has been fixed.
12-08-31 A bug that could cause a core dump after
typeset -m var=.sh.match has been fixed.
12-09-10 Fixed a bug in typeset -m the .sh.match is being renamed.
12-09-07 Fixed a bug in .sh.match code that coud cause the shell
to quitely
13-02-21 The 12-01-16 bug fix prevented .sh.match from being used
in the replacement string. The previous code was restored
and a different fix which prevented .sh.match from being
computed for nested replacement has been used instead.
13-05-28 Fixed two bug for typeset -c and typeset -m for variable
.sh.match.
Changes:
- The SHOPT_2DMATCH option has been removed. This was already the
default behavior previously, and now it's documented in the man
page.
- init.c: Backported the sh_setmatch() rewrite from 93v- 2012-08-24
and 2012-10-04.
- Backported the libast 93v- strngrpmatch() function, as the
.sh.match rewrite requires this API.
- Backported the sh_match regression tests from ksh93v-, with many
other sh_match tests backported from ksh2020. Much of the sh_match
script is based on code from Roland Mainz:
https://marc.info/?l=ast-developers&m=134606574109162&w=2https://marc.info/?l=ast-developers&m=134490505607093
- tests/{substring,treemove}.sh: Backported other relevant .sh.match
fixes, with tests added to the substring and treemove test scripts.
- tests/types.sh: One of the (now reverted) memory leak bugfixes
introduced a CI test failure in this script, so for that test the
error message has been improved.
- string/strmatch.c: The original ksh93v- code for the strngrpmatch()
changes introduced a crash that could occur because strlen would
be used on a null pointer. This has been fixed by avoiding strlen
if the string is null.
One nice side effect of these changes is a considerable performance
improvement in the shbench[1] gsub benchmark (results from 20
iterations with CCFLAGS=-Os):
--------------------------------------------------
name /tmp/ksh-current /tmp/ksh-matchfixes
--------------------------------------------------
gsub.ksh 0.883 [0.822-0.959] 0.457 [0.442-0.505]
--------------------------------------------------
Despite all of the many fixes and improvements in the backported
93v- .sh.match code, there are a few remaining bugs:
- .sh.match is printed with a default [0] subscript (see also
https://github.com/ksh93/ksh/issues/308#issuecomment-1025016088):
$ arch/*/bin/ksh -c 'echo ${!.sh.match}'
.sh.match[0]
This bug appears to have been introduced by the changes from
ksh93v- 2012-08-24.
- The wrong variable name is given for 'parameter not set' errors
(from https://marc.info/?l=ast-developers&m=134489094602596):
$ arch/*/bin/ksh -u
$ x=1234
$ true "${x//~(X)([012])|([345])/}"
$ compound co
$ typeset -m co.array=.sh.match
$ printf "%q\n" "${co.array[2][0]}"
arch/linux.i386-64/bin/ksh: co.array[2][(null)]: parameter not set
- .sh.match leaks out of subshells. Further information and a
reproducer can be found here:
https://marc.info/?l=ast-developers&m=136292897330187
[1]: https://github.com/ksh-community/shbench
When executing a script without a hashbang path like #!/bin/ksh,
ksh forks itself, longjmps back to sh_main(), and then (among other
things) calling sh_reinit() which is the function that tries to
reinitialise as much of the shell as it can. This is its way of
ensuring the child script is run in ksh and not some other shell.
However, this appraoch is incredibly buggy. Among other things,
changes in built-in commands and custom type definitions survived
the reinitialisation, "exporting" variables didn't work properly,
and the hash table and ${.sh.stats} weren't reset. As a result,
depending on what the invoking script did, the invoked script could
easily fail or malfunction.
It is not actually possible to reinitialise the shell correctly,
because some of the shell state is in locally scoped static
variables that cannot simply be reinitialised. There are probably
huge memory leaks with this approach as well. At some point, all
this is going to need a total redesign. Clearly, the only reliable
way involves execve(2) and a start from scratch.
For now though, this seems to fix the known bugs at least. I'm sure
there are more to be discovered.
This commit makes another change: instead of the -h/trackall option
(which has been a no-op for decades), the posix option is now
inherited by the child script. Since there is no hashbang path from
which to decide whether the shell should run in POSIX mode nor not,
the best guess is probably the invoking script's setting.
src/cmd/ksh93/sh/init.c: sh_reinit():
- Keep the SH_INIT state on during the entire procedure.
- Delete remaining non-exported, non-default variables.
- Remove attributes from exported variables. In POSIX mode, remove
all attributes; otherwise, only remove readonly.
- Unset discipline function pointers for variables.
- Delete all custom types.
- Delete all functions and built-ins, then reinitialise the built-ins
table from scatch.
- Free the alias values before clearing the alias table.
- Same with hash table entries (tracked aliases).
- Reset statistics.
- Inherit SH_POSIX instead of SH_TRACKALL.
- Call user init function last, not somewhere in the middle.
src/cmd/ksh93/sh/name.c: sh_envnolocal():
- Be sure to preserve the export attribute of kept variables.
Resolves: https://github.com/ksh93/ksh/issues/350
This bugfix was backported from ksh93v- 2012-10-04. The bug fixed
by this change is one that causes 'typeset -p' to omit the -C flag
when listing compound arrays belonging to a type:
$ typeset -T Foo_t=(compound -a bar)
$ Foo_t baz
$ typeset -p baz.bar
typeset -a baz.bar='' # This should be 'typeset -C -a'
src/cmd/ksh93/sh/nvtype.c:
- Backport change from 93v- 2012-10-04 that sets the array nvalue to
a pointer named Null (which is "") in nv_mktype(), then to Empty
in fixnode().
- Change the Null name from the 93v- code to AltEmpty to avoid
misleading code readers into thinking that it's a null pointer.
src/cmd/ksh93/tests/types.sh:
- Backport the relevant 93v- changes to the types regression tests.
Co-authored-by: Martijn Dekker <martijn@inlv.org>
In ksh93v- 2012-10-04 the following bugfix is noted in the changelog
(this fix was most likely part of ksh93v- 2012-09-27, although that
version is not archived anywhere):
12-09-21 A bug in which the output of a two dimensional sparse
indexed array would cause the second subscript be treated
as an associative array when read back in has been fixed.
Elements that are sparse indexed arrays now are prefixed
type "typeset -a".
Below is a before and after of this change:
# Before
$ typeset -a foo[1][2]=bar
$ typeset -p foo
typeset -a foo=([1]=([2]=bar) )
# After
$ typeset -a foo[1][2]=bar
$ typeset -p foo
typeset -a foo=(typeset -a [1]=([2]=bar) )
src/cmd/ksh93/sh/*.c:
- Backport changes from ksh93v- to print 'typeset -a' before sparse
indexed arrays and properly handle 'typeset -a' in reinput
commands from 'typeset -p'.
src/cmd/ksh93/tests:
- Add two regression tests to arrays.sh for this change.
- Update the existing regression tests for compatibility with the
new printed typeset output.
Special builtins are undeleteable for a reason. But 'enum' and
'typeset -T' allow overriding them, causing an inconsistent state.
@JohnoKing writes:
| The behavior is rather buggy, as it appears to successfully
| override normal builtins but fails to delete the special
| builtins, leading to scenarios where both the original builtin
| and type are run:
|
| $ typeset -T eval=(typeset BAD; typeset TYPE) # This should have failed
| $ eval foo=BAD
| /usr/bin/ksh: eval: line 1: foo: not found
| $ enum trap=(BAD TYPE) # This also should have failed
| $ trap foo=BAD
| /usr/bin/ksh: trap: condition(s) required
| $ enum umask=(BAD TYPE)
| $ umask foo=BAD
| $ echo $foo
| BAD
|
| # Examples of general bugginess
| $ trap bar=TYPE
| /usr/bin/ksh: trap: condition(s) required
| $ echo $bar
| TYPE
| $ eval var=TYPE
| /usr/bin/ksh: eval: line 1: var: not found
| $ echo $var
| TYPE
This commit fixes the following:
The 'enum' and 'typeset -T' commands are no longer allowed to
override and replace special built-in commands, except for type
definition commands previously created by these commands; these
are already (dis)allowed elsewhere.
A command like 'typeset -T foo_t' without any assignments no longer
creates an incompletely defined 'foo_t' built-in comamnd. Instead,
it is now silently ignored for backwards compatibility. This did
have a regression test checking for it, but I'm changing it because
that's just not a valid use case. An incomplete type definition
command does nothing useful and only crashes the shell when run.
src/cmd/ksh93/bltins/enum.c: b_enum():
- Do not allow overriding non-type special built-ins.
src/cmd/ksh93/sh/name.c: nv_setlist():
- Do not allow 'typeset -T' to override non-type special built-ins.
To avoid an inconsistent state, this must be checked for while
processing the assignments list before typeset is really invoked.
src/cmd/ksh93/bltins_typeset.c: b_typeset():
- Only create a type command if sh.envlist is set, i.e., if some
shell assignment(s) were passed to the 'typeset -T' command.
Progresses: https://github.com/ksh93/ksh/issues/350
Building ksh with the tcc (tinycc) compiler failed as of glibc
commit 5d98a7da. The NEWS file in that commit adds:
+* When _DYNAMIC_STACK_SIZE_SOURCE or _GNU_SOURCE are defined,
+ PTHREAD_STACK_MIN is no longer constant and is redefined to
+ sysconf(_SC_THREAD_STACK_MIN).
The tcc build failure seeminly had nothing to do with that --
however, deleting the PTHREAD_STACK_MIN entry and its dependent
THREAD_STACK_MIN entry from conf.tab fixes the build failure.
For reasons unknown, gcc didn't have a problem with it. However,
these config identifiers aren't used anywhere in the ast codebase
(including the full ast-open-history repo) so it should be fine to
just get rid of them; ksh is not and will not be threaded.
NOTE: To build ksh with tcc, you need to build the latest tcc code
from <https://repo.or.cz/tinycc>. The tcc release packages in OS
distributions are typically too old and will not work.
Resolves: https://github.com/ksh93/ksh/issues/437
Thanks to @JohnoKing for the report.
$ unset foo
$ echo ${foo[42]=bar}
(empty line)
Instead of the empty line, 'bar' was expected. As foo[42] was
unset, the conditional assignment should have worked.
$ unset foo
$ : ${foo[42]?error: unset}
(no output)
The expansion should have thrown an error with the given message.
This bug was introduced in ksh 93t 2008-10-01. Thanks to @JohnoKing
for finding the breaking change.
Analysis: The problem was experimenally determined to be in in the
following lines of nv_putsub(). If the array member is unset (i.e.
null), the value is set to the empty string instead:
src/cmd/ksh93/sh/array.c
1250: else
1251: ap->val[size].cp = Empty;
It makes some sense: if there is a value (even an empty one), the
variable is set and these expansions should behave accordingly.
Sure enough, deleting these lines fixes the bug, but at the expense
of introducing a lot of other array-related regressions. So we need
a way to special-case the affected expansions.
Where to do this? If we replace line 1251 with an abort(3) call, we
get this stack trace:
0 libsystem_kernel.dylib __pthread_kill + 10
1 libsystem_pthread.dylib pthread_kill + 284
2 libsystem_c.dylib abort + 127
3 ksh nv_putsub + 1411 (array.c:1255)
4 ksh nv_endsubscript + 940 (array.c:1547)
5 ksh nv_create + 4732 (name.c:1066)
6 ksh nv_open + 1951 (name.c:1425)
7 ksh varsub + 4934 (macro.c:1322)
[rest omitted]
The special-casing needs to be done on line 1250 of array.c, but
flagged in varsub() which processes these expansions. So, varsub()
calls nv_open() calls nv_create() calls nv_endsubscript() calls
nv_putsub(). That's a fairly deep call stack, so passing an extra
flag argument does not seem doable. I did try an approach using a
couple of new bit flags passed via these functions' flags and mode
parameters, but the way this code base uses bit flags is so
intricate, it does not seem to be possible to add or change
anything without unwanted side effects in all sorts of places.
So the only fix I can think of adds yet another global flag
variable for a very special case. It's ugly, but it works.
An elegant fix would probably involve a fairly comprehensive
redesign, which is simply not going to happen.
src/cmd/ksh93/include/shell.h:
- Add global sh.cond_expan flag.
src/cmd/ksh93/sh/array.c: nv_putsub():
- Do not set value to empty string if sh.cond_expan is set.
src/cmd/ksh93/sh/macro.c: varsub():
- Set sh.cond_expan flag while calling nv_open() for one of the
affected expansions.
- Minor refactoring for legibility and to make the fix fit better.
- SSOT: Instead of repeating string "REPLY", use the node's nvname.
- Do not pointlessly add an extra 0 byte when saving id for error
message; sfstruse() already adds this.
Thanks to @oguz-ismail for the bug report.
Resolves: https://github.com/ksh93/ksh/issues/383
Variables with a dot in their name, such as those declared in
namespace { ... } blocks, are usually stored in a separate tree
with their actual names not containing any dots. But under some
circumstances, including at least direct assignment of a
non-preexisting dot variable, dot variables are stored in the main
sh.var_tree with names actually containing dots. With allexport
active, those could end up exported to the environment. This bug
was also present in previous release versions of ksh.
src/cmd/ksh93/sh/name.c: pushnam():
- Check for a dot in the name before pushing a variable to export.
This fix was backported from ksh 93v- 2012-10-04.
src/cmd/ksh93/sh/nvtree.c: nv_outnode():
- If the array is supposed to be empty, do not continue. This
avoids outputting a nonexistent [0]= element for empty arrays.
Resolves: https://github.com/ksh93/ksh/issues/420
Co-authored-by: Martijn Dekker <martijn@inlv.org>
This commit adds a fix for the trap command, backported from a fork
of ksh2020: https://github.com/l0stman/ksh/commit/2033375f
src/cmd/ksh93/sh/jobs.c: job_chldtrap():
- Fixed a use after free bug in the for loop. The string pointed to
by sh.st.trapcom[SIGCHLD] may be freed from memory after
sh_trap(), so it must be reobtained each time sh_trap() is called
from within the for loop.
All variables that are assigned a value should be exported while
the allexport shell option is enabled. This works in most cases,
but variables assigned to with ${var:=foo} or $((var=123)) aren't
exported while allexport is on.
src/cmd/ksh93/sh/name.c:
- nv_putval(): This is the central assignment function; all forms
of variable assignment end up here. So this is the best place
to check for SH_ALLEXPORT and turn on the export attribute.
- nv_setlist(): Remove allexport handling, now redundant.
src/cmd/ksh93/bltins/read.c: sh_readline():
- Remove allexport handling, now redundant.
src/cmd/ksh93/sh/main.c: sh_main():
- nv_putval() is used to initialize PS4 and IFS using nv_putval();
this is after an -a/--allexport specified on the ksh command
line has been processed, so temporarily turn that off.
Co-authored-by: Martijn Dekker <martijn@inlv.org>
This commit implements support for the glibc 2.35
posix_spawn_file_actions_addtcsetpgrp_np(3) extension[2][3],
updating spawnveg(3) to use the new function for setting the
terminal group. This was done with the intention of improving
performance in interactive shells without reintroducing previous
race conditions[4][5].
[1]: https://sourceware.org/pipermail/libc-alpha/2022-February/136040.html
[2]: https://sourceware.org/git/?p=glibc.git;a=commit;h=342cc934
[3]: https://sourceware.org/git/?p=glibc.git;a=commit;h=6289d28d
[4]: https://github.com/ksh93/ksh/issues/79
[5]: https://www.mail-archive.com/ast-developers@research.att.com/msg00717.html
src/cmd/ksh93/sh/path.c:
- Tell spawnveg(3) to set the terminal process group when launching
a child process in an interactive shell.
src/cmd/ksh93/sh/xec.c:
- If posix_spawn_file_actions_addtcsetpgrp_np(3) is available,
allow use of spawnveg(3) (via sh_ntfork()) even with job control
active.
- sh_ntfork(): Reimplement most of the SIGTSTP handling code
removed in commit 66c37202.
src/lib/libast/comp/spawnveg.c,
src/lib/libast/misc/procopen.c,
src/lib/libast/features/sys:
- Add support for posix_spawn_file_actions_addtcsetpgrp_np(3).
- Allow spawnveg to set the terminal process group when pgid == 0.
This was necessary to avoid race conditions when using the new
function.
src/lib/libast/features/lib:
- Detect posix_spawn_file_actions_addtcsetpgrp_np(3).
- Do not detect an OS spawnveg(3). With the API changes to spawnveg
in this pull request ksh probably can't use the OS's spawnveg
function anymore. (That's assuming anything else even provides a
spawnveg function to begin with, which is unlikely.)
src/lib/libast/features/api,
src/cmd/ksh93/include/defs.h:
- Bump libast version (20220101 => 20220201) due to the spawnveg(3)
API change.
src/lib/libast/man/spawnveg.3:
- Document the changes to spawnveg(3) in the corresponding man
page. Currently, it will only use the new tcfd argument if
posix_spawn_file_actions_addtcsetpgrp_np(3) is supported. This
could also be implemented for the fork(2) fallback, but for now
I've avoided changing that since actually using it in the fork
code would likely require a lot of hackery to avoid attempting
tcsetpgrp with vfork (the behavior of tcsetpgrp after vfork is
not portable) and would only benefit systems that don't have
posix_spawn and vfork (I can't recall any off the top of my head
that would fall under that category).
- Updated the man page to account for spawnveg's change in
behavior.
Co-authored-by: Martijn Dekker <martijn@inlv.org>
If the VISUAL or EDITOR environment variable is not set to a value
matching *[Vv][Ii]* or *macs* at initialisation time, then ksh does
not turn on any line editor.
This is user-hostile. New users on Unix-like systems typically have
a simple editor like nano preconfigured as their default, or may
not have the VISUAL or EDITOR variable set at all. So if they try
ksh, they find themselves without basic functionality such as arrow
keys and probably go straight back to bash.
The emacs line editor is by far the most widely used, especially
among new users, so ksh should default to that. Most other shells
already do this.
src/cmd/ksh93/sh/main.c: sh_main():
- On an interactive shell, if on editor was turned on based on
$VISUAL or $EDITOR, turn on emacs before reading input.
Reproducer:
$ ksh -c 'bash -c '\''kill -s INT $$'\''; echo "$?, continuing"'
Expected result: output "258, continuing"; exit status 0.
Actual result: no output; exit status 258. The child process sent
SIGINT only to itself and not to the process group, so the parent
script was wrongly interrupted.
Every shell except ksh93 produces the expected result. ksh93 also
gave the expected result before version 2008-01-31 93s+, which
introduced the code below.
Analysis: The problem is in these lines of code in xec.c,
sh_exec(), TFORK case, parent branch of fork:
1649: if(!sh_isstate(SH_MONITOR))
1650: {
1651: if(!(sh.sigflag[SIGINT]&(SH_SIGFAULT|SH_SIGOFF)))
1652: sh_sigtrap(SIGINT);
1653: sh.trapnote |= SH_SIGIGNORE;
1654: }
[...pipe and I/O handling, wait for command to finish...]
1667: if(!sh_isstate(SH_MONITOR))
1668: {
1669: sh.trapnote &= ~SH_SIGIGNORE;
1700: if(sh.exitval == (SH_EXITSIG|SIGINT))
1701: kill(sh.current_pid,SIGINT);
1702: }
When a user presses Ctrl+C, SIGINT is sent to the entire process
group. If job control is fully off (i.e., !sh_isstate(SH_MONITOR)),
then the process group includes the parent script. Therefore, in a
script such as
$ ksh -c 'bash -c '\''read x'\''; echo "$?, continuing"'
when the user presses Ctrl+C while bash waits for 'read x' input,
the parent ksh script should be interrupted as well.
Now, the code above ignores SIGINT while bash is running. (This is
done using special-casing in sh_fault() to handle that SH_SIGIGNORE
flag for SIGINT.) So, when Ctrl+C interrupts the process group, the
parent script is not getting interrupted as it should.
To compensate for that, the code then detects, using sh.exitval
(the child process' exit status), whether the child process was
killed by SIGINT. If so, it simply assumes that the signal was
meant for the process group including the parent script, so it
reissues SIGINT to itself after unignoring it.
But, as we can see from the broken reproducer above, that
assumption is not valid. Scripts are perfectly free to send SIGINT
to themselves only, and that must work as expected.
src/cmd/ksh93/sh/xec.c: sh_exec(): TFORK: parent branch:
- Instead of ignoring SIGINT, sigblock() it, which delays handling
the signal until sigrelease(). (Note that these are macros
defined in src/cmd/ksh93/features/sigfeatures according to OS
capabilities.)
- This makes reissuing SIGINT redundant, so delete that, which
fixes the bug.
src/cmd/ksh93/sh/fault.c:
- Nothing now sets the SH_SIGIGNORE flag in sh.trapnote, so remove
special-casing added in 2008-01-31 93s+.
Add extra key bindings to the emacs and vi modes
This patch adds the following key bindings to the emacs and vi
editing modes:
- Support for Home key sequences ^[[1~ and ^[[7~ as well as End key
sequences ^[[4~ and ^[[8~.
- Support for arrow key sequences ^[OA, ^[OB, ^[OC and ^[OD.
- Support for the following keyboard shortcuts (if the platform
supports the expected escape sequence):
- Ctrl-Left Arrow: Go back one word
- Alt-Left Arrow: Go back one word (Not supported on Haiku)
- Ctrl-Right Arrow: Go forward one word
- Alt-Right Arrow: Go forward one word (Not supported on Haiku)
- Ctrl-G: Cancel reverse search
- Ctrl-Delete: Delete next word (Not supported on Haiku)
- Added a key binding for the Insert key, which differs in the
emacs and vi editing modes:
- In emacs mode, Insert escapes the next character.
- In vi mode, Insert will switch the editor to insert mode (like
in vim).
src/cmd/ksh93/edit/{emacs,vi}.c:
- Add support for the <M-Left> and <M-Right> sequences. Like in
bash and mksh, these shortcuts move the cursor one word backward
or forward (like the <Ctrl-Left> and <Ctrl-Right> shortcuts).
- Only attempt to process these shortcuts if the escape sequence
begins with $'\E[1;'.
src/cmd/ksh93/edit/vi.c:
- If the shell isn't doing a reverse search, insert the bell
character when Ctrl+G is input.
- Add the Ctrl-Delete shortcut as an alias of 'dw'. Calling
ed_ungetchar twice does not work for 'dw', so Ctrl-Delete was
implemented by using a vp->del_word variable.
Co-authored-by: Martijn Dekker <martijn@inlv.org>
Currently, running ksh under ASan without the ASAN_OPTIONS variable
set to 'detect_leaks=0' usually ends with ASan complaining about a
memory leak in defpathinit() (this leak doesn't grow in a loop, so
no regression test was added to leaks.sh). Reproducer:
$ ENV=/dev/null arch/*/bin/ksh
$ cp -?
cp: invalid option -- '?'
Try 'cp --help' for more information.
$ exit
=================================================================
==225132==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 85 byte(s) in 1 object(s) allocated from:
#0 0x7f6dab42d459 in __interceptor_calloc /build/gcc/src/gcc/libsanitizer/asan/asan_malloc_linux.cpp:154
#1 0x5647b77fe144 in sh_calloc /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/init.c:265
#2 0x5647b788fea9 in path_addcomp /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/path.c:1567
#3 0x5647b78911ed in path_addpath /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/path.c:1705
#4 0x5647b7888e82 in defpathinit /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/path.c:442
#5 0x5647b78869f3 in ondefpath /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/path.c:67
--- cut ---
SUMMARY: AddressSanitizer: 174 byte(s) leaked in 2 allocation(s).
Analysis: The previous code was leaking memory because
defpathinit() returns a pointer from path_addpath(), which has
memory allocated to it in path_addcomp(). This is the code ASan
traced as having allocated memory:
442: return(path_addpath((Pathcomp_t*)0,(defpath),PATH_PATH));
In path_addpath():
1705: first = path_addcomp(first,old,cp,type);
[...]
1729: return(first);
In path_addcomp():
1567: pp = sh_newof((Pathcomp_t*)0,Pathcomp_t,1,len+1);
The ondefpath() function doesn't save a reference to the pointer
returned by defpathinit(), which causes the memory leak:
66: if(!defpath)
67: defpathinit();
The changes in this commit avoid this problem by setting the
defpath variable without also calling path_addpath().
src/cmd/ksh93/sh/path.c:
- Move the code for allocating defpath from defpathinit() into its
own dedicated function called std_path(). This function is called
by defpathinit() and ondefpath() to obtain the current string
stored in the defpath variable. This bugfix is adapted from a
fork of ksh2020: https://github.com/l0stman/ksh/commit/db5c83a6
The array_grow() function calculates the size by multiplying with
sizeof(union Value*), where sizeof(union Value) was clearly meant.
In practice, these are the same size on most (or maybe even all)
systems, as no current member of union Value is larger than a
pointer -- see name.h. But it's still wrong.
Opening the match stack with the STK_SMALL flag causes the stk code
to allocate memory in blocks of 64*sizeof(char*) instead of
1024*sizeof(char*). This caused a significant slowdown which was
exposed by the extglob.ksh module of shbench. Thanks to @JohnoKing
for noticing and reporting the problem.
src/lib/libast/regex/regcomp.c: regcomp():
- Remove STK_SMALL from the stkopen() option bit flags.
Resolves: https://github.com/ksh93/ksh/issues/440
Notable changes:
src/cmd/ksh93/*.c:
- Get rid of all the dtuserdata(FOO,&sh,1) calls backported in
cc492752. These set pointers to sh in Cdt objects. As of
b590a9f1, the code does not use any pointers to sh, so these are
superfluous.
src/cmd/ksh93/data/builtins.c,
src/cmd/ksh93/sh.1:
- As of ksh 93l 2001-06-01, the -h/trackall option has no effect at
all, so trim its documentation.
src/lib/libast/man/stk.3,
src/lib/libast/man/stak.3:
- Correct the documentation on what the ST(A)K_SMALL option bit
actually does based on a reading of the code.
- Document the STK_NULL option bit.
README.md,
src/cmd/ksh93/README:
- Add a note that -fdiagnostics-color=always will break the build.
Ref.: https://github.com/ksh93/ksh/issues/379
src/lib/libast/Mamfile:
- Remove a 'rm -f astmath' command -- a file that is never created.
But on Cygwin this removes astmath.exe, which *is* used. As a
result, executing it failed on Cygwin, so the system incorrectly
detected that Cygwin needs -lm for math functions.
This commit implements support for POSIX_SPAWN_SETSID in
spawnveg(3). The fork/vfork fallback for spawnveg already attempts
to use setsid in the manner described by the man page, so the
posix_spawn implementation should also do so.
src/lib/libast/comp/spawnveg.c:
- Add support for POSIX_SPAWN_SETSID to the posix_spawn version of
spawnveg.
- Minor extra: Remove dead code that's never used (the
_lib_posix_spawn < 2 code block is inside of _lib_posix_spawn >
1, plus when it's manually enabled by changing the previous #if
condition you'll find it causes many regression test failures (at
least on OpenBSD)).
src/lib/libast/man/spawnveg.3:
- Document that spawnveg cannot make the new process a session
leader if the operating system doesn't support POSIX_SPAWN_SETSID
and the new process was created using posix_spawn.
'eval' suffers from the same bug. Reproducer:
$ eval vi
then suspend vi, then try to resume it -- the same as in the
reproducer shown in the previous commit.
src/cmd/ksh93/bltins/misc.c: b_eval():
- Same fix. Do *not* turn off SH_MONITOR.
This fixes yet another whopper of a bug in job control. And it's
been in every version of ksh93 back to 1995, the beginning of
ast-open-archive. <sigh>
This is also bug number 23 that is fixed by simply removing code.
Reproducer:
1. Run vi, or another suspendable program, from a dot script or
POSIX function (ksh handles them the same way). So either:
$ echo vi >v
$ . ./v
or:
$ v() { vi; }
$ v
2. Suspend vi by typing Ctrl+Z.
3. Not one but two jobs are registered:
$ jobs -l
[2] + 85513 Stopped . ./v
[1] - 85512 Stopped . ./v
4. 'fg' does not work on either of them, just printing the job
command name but not resuming the editor. The second job
disappears from the table after 'fg'.
$ fg %2
. ./v
$ fg %2
ksh: fg: no such job
$ fg %1
. ./v
$ fg %1
. ./v
Either way, you're stuck with an unresumable vi that you have to
'kill -9' manually.
src/cmd/ksh93/sh/xec.c: sh_exec(): TFORK:
- Do *not* turn off the SH_MONITOR state flag (which tells ksh to
keep track of jobs) when running an external command from a
profile script or dot script/POSIX function. It's obvious that
this results in an inconsistent job control state as ksh will not
update it when the external command is suspended. The purpose of
this nonsense is surely lost to history as it's been there since
1995 or before. My testing says that removing it doesn't break
anything. If that turns out to be wrong, then the breakage will
have to be fixed in a correct way instead.
On my machine, the build system has been intermittently rebuilding
a sizeable part of libast for no apparent reason. I think I've
finally tracked down the cause: occasionally, the lctab.c file,
generated by port/lcgen.c, randomly changes, triggering said
recompile.
Diff of the latest instance on my system:
--- lctab.c.save 2022-01-28 03:22:47.000000000 +0000
+++ arch/darwin.i386-64/src/lib/libast/lctab.c 2022-01-28 03:30:01.000000000 +0000
@@ -2146,7 +2146,7 @@
#endif
0,0,0,
},
-{"no","norway",LC_primary,
+{"no","norway",0,
#ifdef CTRY_NORWAY
CTRY_NORWAY,
#else
In the port/lc.tab input file, "norway" does *not* have the
"primary" tag, unlike e.g. "sweden" or "united-kingdom". So that
LC_primary value did not belong in the generated code.
A look at the port/lcgen.c code shows that it's using uninitialised
memory. The newof() macro uses malloc, which does not initialise
the memory blocks it allocates:
131:#define newof(p,t,n,x) ((t*)malloc(sizeof(t)*(n)+(x)))
This is then used as follows:
403: case TERRITORY:
404: if (!(tp = newof(0, Territory_t, 1, s - b + 1)))
[...]
444: if (!strcmp(b, "primary"))
445: tp->primary = 1;
The flag is not set to zero if the string does not match, so it's
left uninitialised. Perhaps there are more such problems, but
instead of spending time trying to find them, I'll fix newof().
src/lib/libast/port/lcgen.c:
- In the newof() macro, call calloc(3) instead of malloc(3),
ensuring that all allocated memory is initialised to zero.
posix_spawn(2) was never used as a result of this error as the test
failed to compile, with most systems falling back to vfork(2).
src/lib/libast/features/lib: tst lib_posix_spawn:
- Fix parentheses goof.
Attempting to use array subscript expansion with variables that
aren't set currently causes a spurious syntax error (in ksh93u+ and
older commits the reproducer crashes):
$ ksh -c 'echo ${foo[${bar}..${baz}]}' # Shouldn't print anything
ksh: : arithmetic syntax error
src/cmd/ksh93/sh/macro.c:
- Backport a parser bugfix from ksh93v- 2012-08-24 that avoids
setting mp->dotdot until the copyto() function's loop is
finished.
src/cmd/ksh93/tests/arrays.sh:
- Add regression tests for this bug.
On Cygwin, ksh does not execute scripts without a #! path in a fork
of the ksh process as it does on other systems. Reproducer (run
from ksh):
$ cat test.sh
echo "${BASH_VERSION:-not bash}"
echo "${.sh.version}"
$ chmod +x test.sh
$ ./test.sh
4.4.12(3)-release
./test.sh: line 2: ${.sh.version}: bad substitution
The script was executed in bash instead of ksh.
After this fix, the output on Cygwin is like ksh on other systems:
not bash
Version AJM 93u+m/1.1.0-alpha+dev 2022-01-26
This also fixes a number of regression test failures, as quite a
few tests create and execute temp scripts without a hashbang path.
Analysis: On Cygwin, execve(2) happily executes shell scripts
without a #! path with /bin/sh (which is bash --posix). However,
ksh relies on execve(2) executing binaries or #! only, as it uses
an ENOEXEC failure to decide whether to fork and execute a #!-less
shell script with a reinitialized copy of itself via exscript().
src/cmd/ksh93/sh/path.c: path_spawn():
- Look at the magic first two bytes of the file; if it is "MZ"
(Mark Zbikowski, originator of the .exe format) or "#!", continue
as normal, otherwise simulate an ENOEXEC failure from execve(2)
which will cause ksh to fall back on #!-less script execution.
Notable changes:
src/cmd/ksh93/include/fault.h:
- Get rid of the superflous sh pointer argument in the
sh_pushcontext() and sh_popcontext() macros. (re: 2d3ec8b6)
src/cmd/ksh93/tests/io.sh:
- Tweak a process substitution test to allow up to a second for
unused process substitution processes to disappear. On the Alpine
Linux console (at least the musl libc version), this is needed to
avoid a test failure as long as no GUI is active. As soon as you
start X11, this phenomenon disappears, even on the console. Very
strange, but also very probably not ksh's fault.
src/cmd/ksh93/tests/shtests:
- Instead of just SIGCONT and SIGPIPE, set all signals to default,
just to be sure to avoid spurious test failures due to signals
that were ignored on entry. (It made no difference to the
aforementioned Alpine Linux test failure, so ignored signals had
nothing to do with that -- but still a good idea.)
.github/workflows/ci.yml:
- On the GitHub CI runs, when testing with SHOPTs disabled, disable
SHOPT_SPAWN as well, which tests that everything still works
correctly with the regular fork(2) method.
COPYRIGHT:
- Remove duplicate of BSD license.
src/cmd/ksh93/sh.1:
- Add a new section on history expansion mostly adapted from the
"History substitution" section from the tcsh(1) man page. This
has the standard BSD license which is already in the COPYRIGHT
file. Inapplicable stuff was removed, some new stuff added.
src/cmd/ksh93/edit/hexpand.c,
src/cmd/ksh93/sh/io.c:
- Set '#' as the default history comment character as on bash;
no longer disable it by default.
- Add the 'a' modifier as a synonym for 'g', as on bash.
- Remove pointless static keyword from np variable; since the
value from previous calls is never used it can just be local.
- Use NV_NOADD flag with nv_open() to avoid pointlessly creating
the node if the variable doesn't exist yet.
- Fix a bug in history expansion where the 'p' modifier had no
effect if the 'histverify' option is on.
Reproducer:
$ set -H -o histv
$ true a b c
$ !!:p
$ true a b c▁ <= instead of printed, the line is re-edited
Expected:
$ set -H -o histv
$ true a b c
$ !!:p
true a b c
$ ▁
This is fixed by making 'p' remove the HIST_EVENT bit from the
returned flag bits in hist_expand(), leaving only the HIST_PRINT
flag, then adding special handling for this case to slowread()
in io.c (print the line, then instead of executing it, continue
and read the next line).
Reproducer:
$ set -o histexpand
$ echo foo !#^:h !#^:&
/usr/local/bin/ksh: :&: no previous substitution
ksh(80822,0x10bc2a5c0) malloc: *** error for object 0x10a13bae3: pointer being freed was not allocated
ksh(80822,0x10bc2a5c0) malloc: *** set a breakpoint in malloc_error_break to debug
Abort
Analysis: In hist_expand(), the 'cc' variable has two functions:
it holds a pointer to a malloc'ed copy of the current line, and is
also used as a temporary pointer with functions like strchr().
After that temporary use, it is set to NULL again, because the
'done:' routine checks if it non-NULL to decide whether to free the
pointer. But if an error occurs, the function may jump straight to
'done' without first setting cc to NULL if it had been used as a
temporary pointer. It then tries to free an unallocated pointer.
src/cmd/ksh93/edit/hexpand.c: hist_expand():
- Eliminate this bad practice by using a separate variable for
temporary pointer purposes.
(I was unable to reproduce the crash in a pty regression test,
though it is consistently reproducible in a real interactive
session. So I haven't added that test.)
A 2008 blog post by Finnbar P. Murphy is the only documentation
on these facilities that is available to date. Thankfully, Finnbarr
has graciously granted me permission to use all his ksh93-related
blog posts for ksh 93u+m under the same license as ksh.
Since SHOPT_ACCT (disabled by default) is essentially an older and
more primitive version of SHOPT_AUDIT (enabled by default), we
should probably remove the former in a future release.
src/cmd/ksh93/README-AUDIT.md:
- Added.
src/cmd/ksh93/data/variables.sh: shtab_variables[]:
- Remove unused "CSWIDTH" entry. All use of it (including the
matching CSWIDTHNOD macro) was removed in version 2003-04-22.
src/cmd/ksh93/tests/variables.sh:
- For the tests on the shtab_variables[] variables, read the
variable names straight from variables.c instead of synching
the list in the test script, which would surely be forgotten.
src/cmd/ksh93/tests/*.sh:
- Fix a number of mistaken tries to count errors from a subshell.
- Fix miscellaneous minor breakage and typos.
So far all ksh versions accept event numbers referring to
nonexistent history events in history expansion (-H/-o histexpand),
e.g. !9999 is accepted even if the history file has no item 9999.
These expansions seem to show random content from the history file,
sometimes including binary data. Of course an "event not found"
error should have been thrown instead.
hist_expand() in hexpand.c calls hist_seek() (from history.c)
without any bounds checking except verifying the history event
number is greater than zero. This commit adds a bounds check
to hist_seek() itself as it's called from three other places
in history.c, so perhaps this fixes a few other bugs as well.
src/cmd/ksh93/edit/history.c: hist_seek():
- Use the hist_min() and hist_max() macros provided in history.h
to check bounds. Note that hist_max() yields the number of the
command line currently being entered, so the maximum for seeking
purposes is actually its result minus 1.
History expansion currently crashes under ASan due to a buffer
overflow. Reproducer:
$ set -H
$ !!:s/old/new/
Explanation from <https://github.com/att/ast/issues/1369>:
> The problem is the code assumes the buffer allocated for a string
> stream is zero initialized. But the SFIO code uses malloc() to
> allocate the buffer and does not explicitly initialize it with
> memset(). That it works at all, even without ASAN enabled, is
> purely accidental. It will fail if that malloc() returns a block
> that had been previously allocated, used, and freed. Under ASAN
> the buffer is initialized (at least on my system) to a sequence
> of 0xBE bytes. So the strdup() happily tries to duplicate a
> string that is the size of that buffer and fails when it reads
> past the end of the buffer looking for the terminating zero byte.
src/cmd/ksh93/edit/hexpand.c:
- Backport ksh2020 bugfix that avoids assuming the string stream
has been initialized to zeros:
https://github.com/att/ast/commit/cf16bcca
(minus the incorrect change to the static wm variable).
FreeBSD defines an SF_SYNC macro in sys/socket.h that conflicts
with sfio's SF_SYNC discipline, at best rendering it ineffective.
src/lib/libast/sfio/sfhdr.h:
- Temporarily undef __BSD_VISIBLE while including <sys/socket.h>
to hide the BSD extension with the conflicting definition.
src/lib/libast/features/standards:
- Do not emit #defines for the typ u_long test which is only used
as a heuristic in subsequent tests in this file. (Note that 'set'
can set and unset any iffe command-line --option at runtime.)
- Remove definition of _ISOC99_SOURCE macro. This is another old
GNU thing; feature_test_macros(7) says invoking the compiler with
the option -std=c99 has the same effect. But modern GCC has C11
with GNU extensions as the default, which is fine. If a
particular standard is desired, pass a -std=... flag in $CC.
src/cmd/ksh93/features/rlimits:
- Remove overlooked Linux *64* types/functions hackery.
After defining standards macros it caused a build failure
on at least one version of Void Linux (but not 5.15.14_1).
Thanks to @JohnoKing for the report.
src/cmd/ksh93/sh/subshell.c,
src/lib/libdll/dllnext.c:
- Remove now-redundant local definitions of _GNU_SOURCE and
__EXTENSIONS__ macros.
src/cmd/ksh93/tests/builtins.sh:
- Fix broken sed invocation (re: 41829efa).
The more notable ones are:
src/lib/libast/features/standards:
- Do not redefine _GNU_SOURCE and _FILE_OFFSET_BITS if already
defined from $CCFLAGS. Thanks to @hyanias for the heads-up.
(re: 289dd46c)
src/cmd/ksh93/data/builtins.c,
src/cmd/ksh93/include/shell.h,
src/cmd/ksh93/sh/args.c,
src/cmd/ksh93/sh/name.c:
- Remove -T test code activation option. It was basically unused.
The only thing it did was intentionally introduce a memory leak
in table_unset() if the 4th bit in the option argument was set.
A search in ast-open-history reveals a few more trivial test uses
that were later deleted, but nothing interesting.
src/cmd/ksh93/tests/{basic,path}.sh:
- Skip a couple of tests on AIX avoid hangs, at least one of which
is not ksh's fault. Thanks to @HansH111 for the report.
src/cmd/ksh93/tests/builtins.sh:
- Change one awk use to a more portable sed invocation to placate
systems with ancient awk commands, such as AIX. (re: de795e1f)
Turns out that the standards macros set by features/standards (such
as _GNU_SOURCE for Linux or _DARWIN_SOURCE for macOS) were still
*not* included for most C source files! Instead, they were
selectively included for some files only, sometimes via
FEATURE/standards (the output of features/standards), sometimes
via ast_standards.h which is copied from FEATURE/standards.
Consequently, there were still inconsistencies in the system header
interfaces exposed on Linux, macOS, Solaris, et al. It's no wonder
it sometimes took so much hackery to keep everything building.
Of course, making this consistent had to break things somewhere.
Breakage occurred on 32-bit Linux due to a lot of ugly hackery
involving direct use of internal GNU types like off64_t and
functions like fseek64(). This is now all removed and they are
activated by setting the appropriate feature macro instead, so
these types and functions can be used with their standard names
(off_t, fseek, etc.)
Before committing I've tested these changes on the following
i386/x86_64 systems: Linux (glibc 32 and 64 bit, musl libc 64 bit),
Solaris (32 and 64 bit), illumos (32 and 64 bit), FreeBSD (64 bit),
macOS (64 bit), Cygwin (32 bit), and Haiku (64 bit).
(Note: ast_standards.h is copied from FEATURE/standards, whereas
ast_common.h is copied from FEATURE/common.)
src/lib/libast/include/ast_std.h,
src/lib/libast/stdio/stdhdr.h:
- Include <ast_standards.h> first. This should cause all the AST
and dependent code (such as ksh) to get the standards macros.
src/lib/libast/features/standards:
- For GNU (glibc), #define _FILE_OFFSET_BITS 64 to get large file
support with 64-bit offsets.
- Stop GNU and Cygwin <string.h> form defining the GNU version of
basename(3); on Cygwin, that declaration conflicts with the AST
version (and with POSIX) by using a const char* argument instead
of char*. It is deactivated by defining the macro 'basename' (as
'basename'); this causes GNU string.h to consider it to be
already defined by the standard libgen.h header.
All other changed files:
- Remove direct use of *64* types and functions and a lot of
related hackery.
This commit adds a wrapper for the AIX ar command that uses the
-X64 flag to avoid build errors on that platform.
Resolves: https://github.com/ksh93/ksh/issues/385
Commit 24fc1bbc broke the build on Cygwin in comp/setlocale.c by no
longer defining _GNU_SOURCE on that system in features/standards.
This caused wcwidth() to be hidden by wchar.h though it was
detected in the libraries.
src/lib/libast/features/standards:
- Detect Cygwin along with GNU as a system on which to define
_GNU_SOURCE.
- Add wcwidth() compilation as an extra heuristic to the BSD,
SunOS, Darwin and GNU/Cygwin tests. (Since it's specified as an
optional (X/Open) feature, it should not be tested for in the
generic fallbacks.)
These are minor things I accumulated over the last month or so.
Notable changes:
src/lib/libast/features/api,
src/lib/libast/misc/state.c,
src/lib/libast/comp/conf.tab,
src/cmd/ksh93/include/defs.h:
- Bump internal libast version to 20220101L. We've made a few
additions to the API, at least pathicase (see 71934570, ca3ec200)
and astconf_long (see c2ac69b2), so this should have been done
already. This also updates '/opt/ast/bin/getconf _AST_VERSION'.
- Use AST_VERSION instead of outdated _AST_VERSION.
- In state.c, use AST_VERSION instead of hardcoding the version.
src/cmd/ksh93/sh/xec.c:
- Remove 'restorefd' variable, unused as of 42becab6.
- Remove 'cmdrecurse' function and SH_RUNPROG macro; this was once
used by a few libcmd commands, but ast-open-archive reveals it's
unused as of ast 1999-12-25.
src/cmd/ksh93/sh/*.c:
- Where available, use e_dot instead of "." for consistency; it is
defined as an extern so we might as well use it.
src/cmd/ksh93/tests/*.sh:
- When reporting signal names in fails, include the SIG prefix.
- Fix a broken process hang test in subshell.sh.
src/lib/libast/man/sfdisc.3:
- Removed. The interfaces described here never made it out of AT&T;
they do not exist in any libast version in ast-open-archive.
Resolves: https://github.com/ksh93/ksh/issues/426
In the times(3) fallback for the time keyword (which can be enabled
in xec.c by undefining _lib_getrusage and timeofday), ksh will
print the obtained time incorrectly if TIMEFORMAT is set to use a
precision level of three:
$ TIMEFORMAT=$'\nreal\t%3lR'
$ time sleep .080
real 0m00.008s # Should be '00.080s'
This commit corrects that issue by using 10^precision to get the
correct fractional scaling. Note that the fallback still doesn't
support a true precision level of three (times(3) alone doesn't
support it), so this in effect pads a zero to the end of the output
when the precision level is three.
Additional change to tests/builtins.sh:
- While fixing the above issue I found out that ksh93v- broke
support for passing microseconds to the sleep builtin in the form
of <num>U. I've added a regression test for that bug to ensure it
isn't backported to ksh93u+m by accident.
Co-authored-by: Martijn Dekker <martijn@inlv.org>
The code used to fork subshells when creating/changing aliases will
always fork, even when the alias tree isn't changed:
$ echo $(unalias --man 2> /dev/null; echo $$ ${.sh.pid})
375097 375107
$ alias foo=bar; echo $(alias -p foo; echo $$ ${.sh.pid})
alias foo=bar 375097 375110
This is a bit inefficient, so this commit avoids forking a subshell
unless at least one change is made to the alias table.
src/cmd/ksh93/bltins/typeset.c:
- b_alias(), b_unalias(): Remove sh_subfork() calls.
- setall(): When creating an alias (name contains '='), fork a
virtual subshell before calling nv_open() to add the node.
- unall():
- When unsetting all aliases (-a), fork subshell before dtclear().
- When unsetting one alias, fork subshell before nv_delete().
- Move sh_pushcontext() and sh_popcontext() expansions so that
sh_subfork() is not in between them, as that would cause
program flow corruption or a crash.
Co-authored-by: Martijn Dekker <martijn@inlv.org>
If neither gmacs/emacs nor vi are active, the multiline mode should
not be enabled even if the multiline option is on. Doing so can
cause inconsistent behaviour, particularly in multibyte locales
where, if the shell is compiled with SHOPT_RAWONLY (as is default),
the no-editor mode is actually handled by vi.c.
Also, the new --histreedit and --histverify options only work in
the emacs or vi editors, or in no-editor mode when handled by vi.
Which means they cannot ever work if neither emacs or vi were
compiled in (i.e. SHOPT_ESH and SHOPT_VSH were both disabled).
In that case, there's no point in compiling in those options.
Come to think of it, the same applies to the multiline option.
All changed files:
- Update SHOPT_ESH/SHOPT_VSH preprocessor directives as per above.
src/cmd/ksh93/include/defs.h,
src/cmd/ksh93/include/shell.h:
- Move definitions of history expansion-related options to shell.h,
which is where all the other shell options are defined.
This adds two long-form shell options that modify history expansion
(-H/--histexpand). If --histreedit is on and a history expansion
fails, the command line is reloaded into the next prompt's edit
buffer, allowing corrections. If --histverify is on, the results of
a history expansion are not immediately executed but instead loaded
into the next prompt's edit buffer, allowing further changes.
SH_HISTREEDIT and SH_HISTVERIFY were already handled all along in
slowread() in io.c and via 'reedit' arguments to functions called
there, but could not be turned on as they were only ever exposed as
shopt options in the removed bash compatibility mode (in theory
only, as it failed to compile). I had overlooked them until now.
So, since the code is there, let's expose these options through the
normal long options interface. They're working fine, and activating
them actually makes history expansion tolerable to use.
src/cmd/ksh93/data/options.c:
- Make these options available as "histreedit" and "histverify".
src/cmd/ksh93/data/builtins.c,
src/cmd/ksh93/sh.1:
- Document the "new" options.
src/cmd/ksh93/include/defs.h:
- Remove unused SH_HISTAPPEND and SH_HISTORY2 macros which were
part of the removed bash compatibility code. Note that ksh does
not need a histappend option as it never overwrites the history
file (in the bash mode, this shopt option was a no-op).
The referenced commit introduced a bug that caused command
substitutions to hang, writing infinite zero bytes, when
redirecting standard output on a built-in comand that forks the
command substitution subshell.
The bug was caused by removing the fork when redirecting standard
output in a non-permanent manner. However, simply reintroducing the
fork causes multiple regressions that we had fixed in the meantime.
Thankfully, it looks like this forking workaround is only necessary
when redirecting the output of built-ins. It appears that moving
workaround from io.c to the built-ins handling code in sh_exec() in
xec.c, right before calling sh_redirect(), allows reintroducing the
forking workaround for non-permanent redirections without causing
other regressions.
It would be better if the underlying cause of the hang were fixed
so the workaround becomes unnecessary, but I don't think that is
going to happen any time soon (AT&T didn't manage, either).
src/cmd/ksh93/sh/io.c: sh_redirect():
- Remove forking workaround for redirecting stdout in a comsub.
src/cmd/ksh93/sh/xec.c: sh_exec(): TCOM: built-ins handling code:
- Reimplement the workaround here.
Resolves: https://github.com/ksh93/ksh/issues/416
On Haiku:
# /bin/cat <(echo hi) # no redirection
cat: /tmp/ksh.f29pd8f: No such file or directory
Whereas this works fine:
# /bin/cat < <(echo hi) # with redirection
hi
# /opt/ast/bin/cat <(echo hi) # no redirection; use built-in
hi
Haiku does not have /dev/fd, so uses the FIFO (named pipe) fallback
mechanism. See also: c3eac977
Analysis: In the TFORK part of sh_exec(), forked branch (child),
the FIFO (sh.fifo) is unlinked immediately after opening it. This
is not a problem if the process substitution is used in combination
with a redirection, but if not, then the FIFO is passed on to the
command as a file name argument. This creates a race condition: ksh
was counting on the external 'cat' command opening the FIFO before
the child could unlink it. Whether that race is won depends on
operating system implementation details. When invoking an external
command on Haiku, the race is lost.
src/cmd/ksh93/sh/xec.c: sh_exec(): TFORK: child branch:
- Delay unlinking the FIFO until after executing the process
substitution, when we're about to exit from the child process.
src/cmd/INIT/iffe.sh:
- Remove obsolete $posix_noglob/$posix_glob code to disable and
re-enable pathname expansion. Instead, globally disable it using
'set -o noglob'. Except a couple of 'rm' commands, no aspect of
this code uses globbing (note: 'case' patterns do not count), nor
do any of the shell code feature tests that are eval'ed by iffe.
Field splitting is heavily used via unquoted variable expansions;
globbing should not interfere. (Concrete example: one of the test
notes in src/cmd/ksh93/features/options is "SHOPT_* option probe"
and that SHOPT_* is taken as a file name pattern, so could be
expanded.) For the rm commands, globbing is turned back on in a
subshell. If this breaks something that I've missed, hopefully
that will turn up soon.
- Remove $show/$SHOW code to massage different versions of 'echo'
into acting consistently; instead, use a show_test() function
that uses the POSIX-specified printf command.
- For debugging level 2 and 3, use much more extensive PS4 prompts
taken from modernish (which I wrote), depending on the shell.
- Replace the obsolete 'set -' command. In the ancient Bourne
shell, this disables the v and x options, so is equivalent to
'set +v +x'. This still works on almost all modern shells, but
not yash, and POSIX considers it "unspecified".
- Remove a few echo|sed fallbacks I'd missed (re: 215efa15).
src/cmd/ksh93/features/math.sh:
- Disable pathname expansion here as well.
- Pass down an inherited $IFFEFLAGS to recursive iffe invocations.
Attempting to complete file names in vi mode using tab completion can
fail if the last character on the command line is a space. Reproducer
(note that this bug doesn't occur in emacs mode):
$ set -o vi
$ mkdir '/tmp/foo bar'
$ test -d /tmp/foo\ <Tab>
src/cmd/ksh93/edit/vi.c:
- Don't disable tab completion or reset the tab count just because the
last character on the command line is a space. This bugfix was
backported from ksh93v- 2014-06-06.
src/cmd/ksh93/tests/pty.sh:
- Add a regression test for the tab completion bug.
This commit makes various different improvements to the documentation:
- sh.1: Backported (with changes) mandoc warning fixes from ksh2020
for the ksh93(1) man page: <https://github.com/att/ast/pull/1406>
- Removed unnecessary spaces at the end of lines to fix a few other
mandoc warnings.
- Fixed various typos and capitalization errors in the documentation.
- ANNOUNCE: Document the addition of the ${.sh.pid} variable
(re: 9de65210).
- libast/man/str*: Update the man pages for the libast str* functions
to improve how accurately each function is described.
- ksh93/README: Update regression test/compatibility notes to include
OpenBSD 7.0, FreeBSD 13.0 and WSL running Ubuntu 20.04.
- Change a few places to store the return value from strlen in a
size_t variable rather than signed int.
- comp/setlocale.c: To avoid confusion of two separate variables named
lang, the function local variable has been renamed to langidx.
This commit implements the build fixes required to get ksh running on
Haiku. Note that while ksh does compile, it has a ton of regression test
failures on Haiku.
src/cmd/ksh93/data/signals.c,
src/lib/libast/features/signal.c:
- Add support for the SIGKILLTHR signal, which is supported by BeOS and
Haiku.
- SIGINFO was missing an entry in the libast feature test, so add one
(re: 658bba74).
src/cmd/ksh93/RELEASE:
- Add an entry noting that ksh now compiles on Haiku, albeit with many
regression test failures.
src/cmd/ksh93/{include/terminal.h,sh/path.c}:
- Silence compiler warnings on Haiku.
src/lib/libast/features/mmap:
- The mmap feature test freezes on Haiku, so modify the test to fail
immediately on that OS.
src/lib/libast/misc/signal.c:
- Avoid redefining the signal definition on Haiku to fix a compiler
error.
src/lib/libast/features/nl_types:
- For some reason the nl_item typedef on Haiku doesn't work correctly.
Work around that by creating the nl_item type in the libast nl_types
feature test.
This combines 20 cleanup commits from the dev branch.
All changed files:
- Clean up pointer defererences to sh.
- Remove shp arguments from functions.
Other notable changes:
src/cmd/ksh93/include/shell.h,
src/cmd/ksh93/sh/init.c:
- On second thought, get rid of the function version of
sh_getinterp() as libshell ABI compatibility is moot. We've
already been breaking that by reordering the sh struct, so there
is no way it's going to work without recompiling.
src/cmd/ksh93/sh/name.c:
- De-obfuscate the relationship between nv_scan() and scanfilter().
The former just calls the latter as a static function, there's no
need to do that via a function pointer and void* type conversions.
src/cmd/ksh93/bltins/typeset.c,
src/cmd/ksh93/sh/name.c,
src/cmd/ksh93/sh/nvdisc.c:
- 'struct adata' and 'struct tdata', defined as local struct types
in these files, need to have their first three fields in common,
the first being a pointer to sh. This is because scanfilter() in
name.c accesses these fields via a type conversion. So the sh
field needed to be removed in all three at the same time.
TODO: de-obfuscate: good practice definition via a header file.
src/cmd/ksh93/sh/path.c:
- Naming consistency: reserve the path_ function name prefix for
externs and rename statics with that prefix.
- The default path was sometimes referred to as the standard path.
To use one term, rename std_path to defpath and onstdpath() to
ondefpath().
- De-obfuscate SHOPT_PFSH conditional code by only calling
pf_execve() (was path_pfexecve()) if that is compiled in.
src/cmd/ksh93/include/streval.h,
src/cmd/ksh93/sh/streval.c:
- Rename extern strval() to arith_strval() for consistency.
src/cmd/ksh93/sh/string.c:
- Remove outdated/incorrect isxdigit() fallback; '#ifnded isxdigit'
is not a correct test as isxdigit() is specified as a function.
Plus, it's part of C89/C90 which we now require. (re: ac8991e5)
src/cmd/ksh93/sh/suid_exec.c:
- Replace an incorrect reference to shgd->current_pid with
getpid(); it cannot work as (contrary to its misleading directory
placement) suid_exec is an independent libast program with no
link to ksh or libshell at all. However, no one noticed because
this was in fallback code for ancient systems without
setreuid(2). Since that standard function was specified in POSIX
Issue 4 Version 2 from 1994, we should remove that fallback code
sometime as part of another obsolete code cleanup operation to
avoid further bit rot. (re: 843b546c)
src/cmd/ksh93/bltins/print.c: genformat():
- Remove preformat[] which was always empty and had no effect.
src/cmd/ksh93/shell.3:
- Minor copy-edit.
- Remove documentation for nonexistent sh.infile_name. A search
through ast-open-archive[*] reveals this never existed at all.
- Document sh.savexit (== $?).
src/cmd/ksh93/shell.3,
src/cmd/ksh93/include/shell.h,
src/cmd/ksh93/sh/init.c:
- Remove sh.gd/shgd; this is now unused and was never documented
or exposed in the shell.h public interface.
- sh_sigcheck() was documented in shell.3 as taking no arguments
whereas in the actual code it took a shp argument. I decided to
go with the documentation.
- That leaves sh_parse() as the only documented function that still
takes an shp argument. I'm just going to go ahead and remove it
for consistency, reverting sh_parse() to its pre-2003 spec.
- Remove undocumented/unused sh_bltin_tree() function which simply
returned sh.bltin_tree.
- Bump SH_VERSION to 20220106.
(nmake makefiles) defined this macro:
__OBSOLETE__ == $("6 months ago":@F=%(%Y0101)T)
This was used to automatically disable code after a period between
6 and 18 months, on 1st Jan of each year, in preprocessor
directives like:
#if __OBSOLETE__ < 20080101
// obsolete code here
#endif
However, when compiling without nmake (as we do), this __OBSOLETE__
macro is not defined at all. And undefined macros evaluate to zero
in arithmetic comparisons, so all that obsolete code has been
getting compiled. Thankfully it doesn't seem to have done any harm,
but all that code was supposed to expire between 2008 and 2014.
src/lib/libast/disc/sfstrtmp.c:
- Removed. Was supposed to be a stub #if __OBSOLETE__ >= 20070101.
src/lib/libast/include/ast.h:
- Remove unused fmtbasell() macro (/* until 2014-01-01 */).
Other changed files:
- Remove __OBSOLETE__d code.
bin/package, src/cmd/INIT/package.sh:
- Automatically update an existing flat view even if 'flat' wasn't
given for a make action. This stops a flat view becoming
inaccurate if you forget to use 'bin/package flat make'
consistently. If the $PACKAGEROOT/lib/package/gen directory
exists, an existing flat view is assumed.
- Correct symlink fallbacks. We need an absolute path for the
symlink target or it's going to be broken.
.gitignore:
- Update.
This fixes the use after free issue that caused typeset -m to crash on
older versions of OpenBSD and under ASan. The problem that was causing
the failure was that the ap pointer wasn't set to null after the memory
associated with it was freed. This commit backports a bugfix from
ksh93v- 2013-06-28 that sets ap to null before freeing the associated
memory and adds a check that makes sure ap is still a valid pointer
before calling array_unscope().
tests/types.sh changes:
- Avoid redirecting stderr to /dev/null, as this test shouldn't print
anything to stderr.
- Apply error message improvement from
https://github.com/ksh93/ksh/issues/231#issue-834252084.
tests/arrays.sh change:
- Apply error message improvement from
https://github.com/ksh93/ksh/issues/229#issue-834240645 (re: 7c7fde75).
Resolves: https://github.com/ksh93/ksh/issues/231
This backports a ksh2020 fix for an ASan heap-use-after-free error in
arrays.sh. The arrays regression tests were failing under ASan because
the ap pointer was used after the memory allocated to it was freed by
_nv_unset(). ksh2020 commit:
f1e5119e31
In backtick command substitutions '\\' will only print one backslash,
while placing a backslash in front of a different character will
print a backslash followed by that character:
$ echo `echo '\\\\ \3'`
\\ \3
$ echo $(echo '\\\\ \3')
\\\\ \3
This difference in behavior was causing iffe to fail on Haiku after
the changes in aeda3502. This commit applies fixes for backslashes
in quoted strings to fix the Haiku build failure and other possible
bugs introduced by the referenced commit.
bin/package, src/cmd/INIT/package.sh:
- Make the EXIT (0) trap report failure based on $error_status
(see d18469d6), replacing 'done' with 'failed' if it is nonzero.
- Remove extra space before 'at' in that report line.
- If we get a signal, we have to set error_status to nonzero
manually so that the correct exit message is printed.
I've reconsidered excluding build system internals (and also '*.o'
files) from the flat view. Really the only thing that should be
excluded are *.old files.
bin/package, src/cmd/INIT/package.sh:
- Do not exclude anything except *.old files from the flat view.
- This would delete bin/package when cleaning up the flat view,
so explicitly keep bin/package in the clean routine.
- Be much faster about updating an existing flat view by checking
if a link already exists before creating one.
- Add silent cleanup for dozens of orphaned macOS *.dSYM bundles
belonging to deleted temporary feature test executables.
src/cmd/INIT/{iffe,ignore,silent}.sh, bin/{ignore,silent}:
- Remove obsolete Bourne shell fallbacks.
- Modernise command substitutions.
- Remove unused literal() function.
- Update copy() function to use printf.
- Distinguish just two shell types now: ksh and POSIX.
- compile(): Remove obsolete/incorrect grepping for signal messages.
Add a POSIX-compiliant check with 'kill -l' to see if the
compiler was terminated by a signal.
This upstreams the patch to init.c that is necessary to build dtksh
(graphical extensions for CDE, see <https://cdesktopenv.sf.net/>).
It has no effect when building regular ksh. Upstreaming it avoids
the need to keep updating it when changes to init.c are made.
I called the flat view featuritis, but it turns out the dtksh build
system depends on it. But it was broken, so let's make a proper
version instead. This new approach does not symlink directories
before make, but hardlinks files after make. To make performance
tolerable, it requires a modern POSIX 'find' utility with '{} +'.
bin/package, src/cmd/INIT/package.sh:
- We're going to depend on 'test X -ef Y' to check if X is the same
file as Y, so add that to the $min_posix checks even though it is
not technically POSIX. But it's so close to universally available
it doesn't really make a difference.
- After 'make', create a flat view by hardlinking arch/$HOSTTYPE
files, minus build system internals, onto their corresponding
paths in $PACKAGEROOT. Fall back to symlinking if hardlinking
fails (this would happen when crossing device boundaries).
- Clean up flat view when doing clean/clobber. This is done by
using 'find' to loop through the files in arch/ again and
removing any root paths that are the same file as their
corresponding arch/ path (this is where test X -ef Y comes in).
Then delete arch/$HOSTTYPE, then delete empty directories left.
- Check for the 'flat' qualifier when doing clean/clobber. If
given, do not delete arch/$HOSTTYPE but only clean up the flat
view and remove empty directories.
src/cmd/INIT/Mamfile:
- Do not create the lib/package directory. (re: beb3c64a)
.gitignore:
- Update.
After further examining the iffe code I found that fail{ ... }end
as well as pass{ ... }end blocks are executed in iffe's current
environment, using a simple 'eval', with no safeguards whatsoever!
This does of course afford maximum flexibility... for example, a
block can 'exit 1' to abort the iffe run and the whole build with
it. We can use this to abort properly on fatal compilation errors.
src/cmd/INIT/iffe.sh:
- Complete the fail{ and pass{ documentation; it should really be
known that they run in iffe's current environment.
- Make one change: for just the 'eval' command that runs these
blocks, redirect standard error back to the saved $stderr file
descriptor so these blocks can write error messages using the
standard >&2 instead of the undocumented >&$stderr.
src/lib/**/features/*:
- Write error message to standard error and error out properly when
an output{ ... }end block fails to compile, instead of writing an
#error directive to error out later.
As observed previously (see 3654ee73, 7e6bbf85, 79d19458), the ksh
93u+ codebase on which we rebased development was in a transition:
AT&T evidently wanted to make it possible to have several shell
interpreter states in the same process, which in theory would have
made it possible to start a complete new shell (not just a
subshell) without forking a new process.
This required transitioning from accessing the 'sh' state struct
directly to accessing it via pointers (usually but not always
called 'shp'), introducing a lot of bug-prone passing around of
those pointers via function arguments and other state structs.
Some of the original 'sh' struct was separated into a 'struct
shared' called 'shgd' a.k.a. 'sh.gd' (global data) instead; these
were global state variables that were going to be shared between
the different main shell environments sharing a process. Yet, for
some reason, that struct was allocated dynamically once at init
time, requiring yet another pointer to access it. <shrug>
None of this ever worked, because that transition was incomplete.
It was much further along in the ksh 93v- beta, but I don't think
it actually worked there either (not very much really did). So,
starting a new shell has always required starting a new process.
So, now that it's clear what they were trying to do, should we try
to make it work? I'm going to go with a firm "no" on that question.
Even non-forking (virtual) subshells, something quite a bit less
ambitious, were already an unmitigated nightmare of bugs. In 93u+m
we fixed a load of bugs related to those, but I'm sure there are
still many left. At the very least there are multiple memory leaks.
I think the ambition to go even further and have complete shells
running separate programs share a process, particularly given the
brittle and buggy state of the existing codebase, is evidence that
the AT&T team, in the final years, had well and truly lost the
ability to think "wait a minute, aren't we in over our heads here,
and why are we doing this again? Is this *actually* a feasible and
useful idea?"
In my view, having entirely separate programs share a process is a
*terrible*, horrible, no-good idea that takes us back to the bad
old days before Unix, when kernels and CPUs were unable to enforce
any memory access restrictions. Programmers are imperfect. If
you're going to run a new program, you need the kernel to enforce
the separation between programs, or you're just asking for memory
corruption and security holes. And that separation is enforced by
starting a new program in a new process. That's what processes are
*for*. And if you need *that* to be radically performance-optimised
then you're probably doing it wrong anyway.
(By the way, I would still argue the same for subshells, even after
we fixed many bugs in virtual subshells. But forking all subshells
would in fact cause many scripts to slow down, and the community
would surely revolt. <sigh> Maybe I should make it a shell option
instead, so scripts can 'set -o subfork' for reliability.)
It is also unclear how they were going to make something like
'ulimit' work, which can only work in a separate process. There
was no sign of a mechanism to fork a separate program's shell
mid-execution like there is for subshells (sh_subfork()).
Anyway... I had already changed some code here and there to access
the sh state struct directly, but as of this commit I'm beginning
to properly undo this exercise in pointlessness. From now on, we're
exercising pointerlessness instead.
I'll do this in stages to make any problems introduced more
traceable. Stage 0 restores the full 'sh' state struct to its
former static glory and reverts 'shgd' as a separate entity.
src/cmd/ksh93/sh/defs.c,
src/cmd/ksh93/include/defs.h,
src/cmd/ksh93/include/shell.h
src/cmd/ksh93/Mamfile::
- Move 'struct sh_scoped' and 'struct limits' from defs.h to
shell.h as the sh struct will need their complete definitions.
- Get rid of 'struct shared' (shgd) in defs.h; its members are
folded back into their original place, the main Shell_t struct
(sh) in shell.h. There are no name conflicts.
- Get rid of the _SH_PRIVATE macro in defs.h. The members it
defines are now defined normally in the main Shell_t struct (sh)
in shell.h.
- To make this possible, move <history.h> and "fault.h" includes
from defs.h to shell.h and update the Mamfile accordingly.
- Turn sh_getinterp() and shgd into macros that resolve to (&sh).
This will allow the compiler to optimise out many pointer
dereferences already.
- Keep extern sh_getinterp() for libshell ABI compatibility.
src/cmd/ksh93/sh/init.c:
- sh_init(): Do not calloc (sh_newof) the sh or shgd structs.
- sh_getinterp(): Keep function for libshell ABI compat.
After 'unset CDPATH', CDPATH continued to work as if nothing
happened. Unsetting it should be a valid way to deactivate it.
This bug is in every ksh93 version.
src/cmd/ksh93/bltins/cd_pwd.c: b_cd():
- Fix a manifest logic error: first check if CDPATH (CDPNOD) is
unset before assigning to 'cdpath', not the other way around.
Setting the 'cdpath' pointer is what activates the CDPATH search.
That intermittent regression test failure in types.sh seems to be
gone. So let's reimport the regex changes into the 1.0 branch to
subject them to wider testing and make sure any failures stay gone.
(re: 48568476, 38aab428, 1aa8f771)
[Original commit message from 1aa8f771 follows]
There are two main changes:
1. The regex code now creates and uses its own stack (env->mst)
instead of using the shared standard stack (stkstd). That seems
likely to be a good thing.
2. Missing mbinit() calls were inserted. The 93v- code uses a
completely different multibyte characters API, so these needed
to be translated back to the older API. But, as mbinit() is no
longer a no-op as of 300cd199, these calls do stop things from
breaking if a previous operation is interrupted mid-character.
I think there might be a couple of off-by-one errors fixed as well,
as there are two instances of this change:
- while ((index += skip[buf[index]]) < mid);
+ while (index < mid)
+ index += skip[buf[index]];
This commit ports performance optimizations from illumos for the libsum
code (used by the cksum and sum builtins):
98bea71f0d
The new codepath in libsum uses prefetching and loop unrolling to
improve performance (prefetching is done with __builtin_prefetch()
or sun_prefetch_read_many() if either is available).
Script for testing (note that cksum must be enabled in
src/cmd/ksh93/data/builtins.c):
#!/bin/ksh
builtin cksum || exit 1
for ((i=0; i!=50000; i++)) do
cksum -x att /etc/hosts
done >/dev/null
Results on Linux x86_64 (using CCFLAGS=-O2):
$ echo 'UNPATCHED:'; time arch/linux.i386-64/bin/ksh /tmp/foo; echo 'PATCHED'; time /tmp/ksh /tmp/foo
UNPATCHED:
real 0m09.989s
user 0m07.582s
sys 0m02.406s
PATCHED:
real 0m06.536s
user 0m04.331s
sys 0m02.204s
src/lib/libsum/{sum-att.c,sum-crc.c,Mamfile}:
- Port the performance optimizations from illumos to 93u+m libsum. To
prevent problems with older versions of GCC, avoid the new codepath
if GCC is older than the 3.1 release series. Additionally, the ast.h
header must be included to handle tcc defining __GNUC__ on FreeBSD.
- Apply some build fixes to allow the new codepath to build with Clang
3.6 and newer (my own testing indicates an even better performance
improvement with Clang than with GCC).
This patch adds a few extra options to the ulimit command (if the OS
supports them). These options are also present in Bash, although in ksh
additional long forms of each option are available:
ulimit -k/--kqueues This is the maximum number of kqueues.
ulimit -P/--npts This is the maximum number of pseudo-terminals.
ulimit -R/--rttime This is the time a real-time process can run
before blocking, in microseconds. When the
limit is exceeded, the process is sent SIGXCPU.
Other changes:
- bltins/ulimit.c: Change the formatting from sfprintf and increase the
size of the tmp buffer to prevent text from being cut off in ulimit
-a (this was required to add ulimit -R).
- data/limits.c: Add support for using microseconds as a unit.
This commit primarily makes changes that allow the regression tests to
run without any of the /opt/ast/bin builtins compiled into ksh. It also
makes a minor improvement to one of the tests in locale.sh by
shellquoting an error message.
Co-authored-by: Martijn Dekker <martijn@inlv.org>
The unused mkservice and eloop builtins are currently not built, and if
an attempt to compile them is made the build ends in failure. This
commit backports a few build fixes from ksh93v- 2012-08-24 that allow
mkservice and eloop to build (plus an additional compiler warning fix
not in ksh93v-). I've also added a new SHOPT_MKSERVICE setting (turned
off by default) so that mkservice and eloop can be built if the user
chooses to include them in their build of ksh.
On modern operating systems, memory management is non-deterministic
(i.e. random, unpredictable) to varying degrees. This makes testing
for memory leaks a nightmare as the OS may decide to randomly grow
a process's memory allocation at any time for no apparent reason,
causing intermittent test failures that do not represent real
memory leaks. So far, the leaks test tried to cope with this by
using a large number of iterations plus a certain amount of bytes
of tolerance per iteration. This was inefficient and on some
systems still did not fully eliminate intermittent test failures.
This commit introduces a new testing algorithm that is designed to
cope with a large degree of unpredictability. Instead of a fixed
number of test iterations, it defines a maximum (16384), dividing
them in blocks of 128 iterations. It also defines a minimum number
of sequential "good" iteration blocks, counted if memory usage did
not increase from one block to the next. That minimum number is set
to 16. The theory is that if we can get 16 "good" iteration blocks
in a row, we can safely assume it's not a real memory leak, break
the loop, and consider the test succeeded. That "good" sequence is
allowed to occur at any point in the loop, creating a high built-in
tolerance for non-deterministic shenanigans. It also speeds up the
tests, as successful tests can bow out at 16 * 128 == 2048
iterations if they're lucky. If the OS decides to randomly grow the
memory heap, it may take more tries, but almost (?) certainly not
more than the maximum 16384 (128 blocks). If the counter reaches
that, then we assume a memory leak and throw a test failure.
We're also no longer testing with byte granularity in any case; the
randomness of memory management makes that pointless. All getmem()
function versions now return kibibytes (1024 bytes).
This should eliminate the need for workarounds such as initial
iterations to "steady the state" or a tolerance of a certain number
of bytes. I've experimentally determined the exact values
(max_iter, block_iter, min_good_blocks) that seem to work reliably
on all systems I've tested. They are easy to tweak if necessary.
To make all this manageable, this commit hides all the supporting
code in a triplet of aliases (TEST, DO, DONE) that, when used
correctly, create a grammatically robust shell code block: you can
add redirections, pipe into it, etc. as expected. This makes the
actual tests a great deal easier to read as well.
src/cmd/ksh93/tests/pty.sh:
- Implement new leaks testing framework as described and convert
all the tests to it.
- Mark known leaks with a 'known' variable. Print non-fail warnings
for all known leaks, but skip the tests by default. Test them
only if DEBUG is exported. This is better than commenting them
out as we will no longer be tempted to forget about these.
- Move the test for large command substitutions to subshell.sh --
it's not in fact a leak test; instead, it checks that command
substitutions don't lose data.
src/cmd/ksh93/tests/_common: err_exit():
- Since we're printing more warnings, clearly mark all test
failures with 'FAIL:' to make them stand out.
src/cmd/ksh93/tests/shtests.
src/cmd/ksh93/tests/pty.sh:
- Special-case leaks.sh for counting tests; grep ^TEST.
- Special-case pty.sh as well while we're at it by grepping tst()
calls. Remove all the dummy '# err_exit #' comments from pty.sh
as they are now no longer used for counting the tests.
The variable 'i' had already been used for a non-numeric purpose,
so when declaring it integer in a subshell it's necessary to
initialise it with value 0 or an arithmetic error is shown (which
does not interrupt the test or make it fail).
For the record, the errpr was mine, not Johnothan's.
This commit backports the whence '-t' option from ksh93v-. The '-t'
option is useful when one needs to identify the type of a command.
The '-t' flag was added by ksh93v- for compatibility with Bash.
It should be noted the ksh93v- patch had one bug, which this commit
fixes. Path-bound builtins from /opt/ast/bin were classified as
files if loaded from /opt/ast/bin in the PATH. Reproducer:
$ PATH=/opt/ast/bin whence -t cat
file
src/cmd/ksh93/bltins/whence.c:
- Simplify the bitmask values for the command and whence builtin
flags.
- Add the -t flag to the whence and type builtins. To prevent bugs,
-t will always override -v if both of those flags were passed.
src/cmd/ksh93/data/builtins.c,
src/cmd/ksh93/sh.1:
- Add documentation for the new -t option.
I'd forgotten to check for uses of the __STDC__ macro. This is
defined on all C compilers that support C89/C90 or later standards.
So we can remove all fallback code disabled by that macro.
Depending on the OS, the heredoc.sh regression tests, and possibly
others, still crashed with the -x option (xtrace) on.
Analysis: The lexer crashes in lex_advance(). Something has caused
an inconsistent lexer state, and it happened earlier on, so the
backtrace is useless for figuring out where that happened.
But I think I've found it. It's the sh_mactry() call here:
src/cmd/ksh93/sh/xec.c, lines 2800 to 2807 in f7213f03
2800: if(!(cp=nv_getval(sh_scoped(shp,PS4NOD))))
2801: cp = "+ ";
2802: else
2803: {
2804: sh_offoption(SH_XTRACE);
2805: cp = sh_mactry(shp,cp);
2806: sh_onoption(SH_XTRACE);
2807: }
sh_mactry() needs to parse the contents of $PS4 to perform
expansions and command substitutions in it, which involves the
lexer. If that happens in a here-document, the lexer is in the C
function call stack, in the middle of parsing the here-document.
Result: inconsistent lexer state. Solution: save and restore lexer
state in sh_mactry().
After this commit, all regression tests should pass with the
'-x'/'--xtrace' option in use, with no errors or crashes.
Note for backporters: this fix depends both on on d7cada7b and on
the consistency fix for the Lex_t type's size applied in a7ed5d9f.
src/cmd/ksh93/include/shlex.h:
- Cosmetic fix: remove a copied & pasted backslash. (re: a7ed5d9f)
src/cmd/ksh93/sh/macro.c: sh_mactry():
- Save and restore the lexer state before letting sh_mactrim()
indirectly parse and execute code.
src/cmd/ksh93/tests/*.sh:
- Turn off xtrace in various command substitutions that contain
2>&1 redirections, so that the xtrace output is not caught by
the command substitutions, causing tests to fail incorrectly.
- Turn off xtrace for a few code blocks with 2>&1 redirections,
stopping xtrace output from being written to standard output.
Resolves: https://github.com/ksh93/ksh/issues/306 (again)
On some systems (at least Linux and macOS):
1. Run on a command line: t=$(sleep 10|while :; do :; done)
2. Press Ctrl+C in the first 10 seconds.
3. Execute any other command substitution. The shell crashes.
Analysis: Something in the job_wait() call in the sh_subshell()
restore routine may be interrupted by a signal such as SIGINT on
Linux and macOS. Exactly what that interruptible thing is remains
to be determined. In any case, since job_wait() was invoked after
sh_popcontext(), interrupting it caused the sh_subshell() restore
routine to be aborted, resulting in an inconsistent state of the
shell. The fix is to sh_popcontext() at a later stage instead.
src/cmd/ksh93/sh/subshell.c: sh_subshell():
- Rename struct checkpt buff to checkpoint because it's clearer.
- Move the sh_popcontext() call to near the end, just after
decreasing the subshell level counters and restoring the global
subshell data struct to its parent. This seems like a logical
place for it and could allow other things to be interrupted, too.
- Get rid of the if(shp->subshell) because it is known that the
value is > 0 at this point.
- The short exit routine run if the subshell forked now needs a new
sh_popcontext() call, because this is handled before restoring
the virtual subshell state.
- While we're here, do a little more detransitioning from all those
pointless shp pointers.
Fixes: https://github.com/ksh93/ksh/issues/397
This commit was originally intended to fix just one bug with shcomp's
handling of 'alias -p', but while fixing that I found a large number
of related issues in the alias command's -p, -t and -x options. The
current patch provides bugfixes for all of the bugs listed below:
1) Listing aliases in a script with 'alias -p' or 'alias' broke
shcomp's bytecode output:
https://github.com/ksh93/ksh/issues/87#issuecomment-813819122
2) Listing individual aliases with the -p option doesn't work:
$ alias foo=bar bar=foo
$ alias foo
foo=bar
$ alias -p foo # No output
3) Listing specific tracked aliases with -pt does not display them
in a reusable format, but rather adds another tracked alias:
$ hash -r cat vi
$ alias -pt vi # No output
$ alias -pt rm
$ alias -t
cat=/usr/bin/cat
rm=/usr/bin/rm
vi=/usr/bin/vi
4) Listing all tracked aliases with -pt does not output them in a
reusable format (the resulting command printed only creates a
normal alias, which is different from a tracked alias):
$ hash -r cat
$ alias -pt
alias cat=/usr/bin/cat # Expected 'alias -t cat'
5) Listing a non-existent alias with -p doesn't cause an error:
$ unalias -a
$ alias -p notanalias # No output
$ echo $?
0
$ alias notanalias
notanalias: alias not found
$ echo $?
1
$ hash -r
$ alias -pt notacommand # No output
$ echo $?
0
6) Attempting to list 256 non-existent aliases results in exit
status zero:
$ unalias -a
$ alias $(awk -v ORS= 'BEGIN { for(i=0;i<256;i++) print "x "; }')
x: alias not found
--cut error message--
$ echo $?
0
Changes:
- typeset.c: Avoid printing anything while shcomp is compiling a
script. This is needed because the alias command is run by shcomp
to prevent parsing issues.
- b_alias(): To avoid adding tracked aliases with -pt, set
tdata.aflag to '+' so that setall() and other related functions
only list tracked aliases.
- b_alias(): Set tdata.pflag to 1 so that setall() and other
functions recognize -p was passed.
- print_value(): Add support for listing specific aliases with
'alias -p'.
- setall(): To avoid any issues with zombie tracked aliases (see also
the regression tests) ignore tracked alias nodes marked with the
NV_NOALIAS attribute. This bit is set for tracked alias nodes by
the nv_rehash() function.
- setall(): For backward compatibility, continue incrementing the
exit status for each invalid alias and tracked alias passed. This
was already how alias behaved when listing aliases without -p, so
using -p shouldn't cause a change in behavior:
$ unalias -a
$ alias foo bar
foo: alias not found
bar: alias not found
$ echo $?
2
To fix bug 6, the exit status is set to one if an enforced 8-bit
exit status would be zero.
- print_namval(): Set the prefix to 'alias -t' so that listing
tracked aliases with 'alias -pt' works correctly.
- data/msg.c and include/name.h: Add an error message for when
'alias -pt' doesn't find a tracked alias.
- tests/alias.sh: Add a ton of regression tests for the bugs fixed in
this commit.
The sh_close() function fails to set errno to EBADF when passed a
negative (invalid) file descriptor. This commit fixes the issue
by setting errno if the file descriptor is a negative value
(backported from ksh93v- 2012-08-24).
This takes another step towards cleaning up the build system. We
now do not even pretend to be theoretically compatible with
pre-1989 K&R C compilers or with C++ compilers. In practice, this
had already been broken for many years due to bit rot.
Commit 46593a89 already removed the license handling enormity that
depended on proto, so now we can cleanly remove it altogether. But
we do need to leave some backwards compatibility stubs to keep the
build system compatible with older AST code; it should remain
possible to build older ksh versions with the current build system
(the bin/ and src/cmd/INIT/ directories) for testing purposes.
So as of now there is no more __MANGLE__d rubbish in your generated
header files. This is only about a quarter of a century overdue...
This commit also includes a huge amount of code cleanup to remove
thousands of unused K&R C fallbacks and other cruft, particularly
in libast. This code base should now be a little easier to
understand for people who are familiar with a modern(ish) C
standard.
ratz is now also removed; this was a standalone and simplified 2005
version of gunzip. As of 6137b99a, none of our code uses it, even
theoretically. And the real g(un)zip is now everywhere.
src/cmd/INIT/proto.c, src/cmd/INIT/ratz.c:
- Removed.
COPYRIGHT:
- Remove zlib license; this only applied to ratz.
bin/package, src/cmd/INIT/package.sh:
- Related cleanups.
- Unset LC_ALL before invoking a new shell, respecting the user's
locale again and avoiding multibyte character corruption on the
command line.
src/cmd/INIT/proto.sh:
- Add stub for backwards compatibility with Mamfiles that depend on
proto. It does nothing but pass input without modification and is
now installed as the new arch/*/bin/proto by src/cmd/INIT/Mamfile.
src/cmd/INIT/iffe.sh:
- Ignore the proto-related -e (--package) and -p (--prototyped)
options; keep parsing them for backwards compatibility.
- Trim the macros passed to every test to their standard C
versions, removing K&R C and C++ versions. These are now
considered to be for backwards compatibility only.
src/cmd/INIT/iffe.tst:
- Remove proto(1) mangling code.
By the way, iffe can be regression-tested as follows:
$ bin/package use # set up environment in a child shell
$ regress src/cmd/INIT/iffe.tst
$ exit # leave package environment
src/cmd/INIT/make.probe, src/cmd/INIT/probe.win32:
- Remove code to handle C++.
src/lib/libast/features/common:
- As in iffe.sh above, trim macros designed for compatibility with
C++ and ancient C compilers to their standard C versions and
comment that they are for backwards compatibility with AST code.
This is needed to keep all the old ast and ksh code compiling.
src/cmd/ksh93/sh/init.c,
src/cmd/ksh93/sh/name.c:
- Clarify libshell ABI compatibility function versions of macros.
A "proto workaround" comment in the original code mislead me into
thinking this had something to do with the removed proto(1), but
it's unrelated. Call the workaround macro BYPASS_MACRO instead.
src/cmd/ksh93/include/defs.h:
- sh_sigcheck() macro: allow &sh as an argument: parenthesise shp.
src/cmd/ksh93/sh/nvtype.c:
- Remove unused nv_mkstruct() function. (re: d0a5cab1)
**/features/*:
- Remove obsolete iffe 'set prototyped' option.
**/Mamfile:
- Remove all references to the ast/prototyped.h header.
- Remove all use of the proto command. Simply copy instead.
*** 850-ish source files: ***
- Remove all '#pragma prototyped' directives.
- Remove all C++ compat code conditional upon defined(__cplusplus).
- Remove all use of the _ARG_ macro, which on standard C expands to
its argument:
#define _ARG_(x) x
(on K&R C, it expanded to nothing)
- Remove all use of _BEGIN_EXTERNS_ and _END_EXTERNS_ macros (empty
on standard C; this was for C++ compatibility)
- Reduce all #if __STD_C (standard code) #else (K&R code) #endif
blocks to the standard code only, without use of the macro.
- Same for _STD_ macro which seems to have had the same function.
- Change all instances of 'Void_t' to standard 'void'.
