cde/NEWS

This documents significant changes in the 93u+m branch of AT&T ksh93.
For full details, see the git log at: https://github.com/ksh93/ksh

Any uppercase BUG_* names are modernish shell bug IDs.

2020-09-04:

- Fixed a bug that caused a syntax error to be thrown if the special parameter
  expansions ${!} and ${$} (including braces) were used within a here-document.
  Bug reported by @Saikiran-m on GitHub.

2020-09-01:

- The bash-style '&>file' redirection shorthand (for '>file 2>&1') is now
  always recognised and not only when running rc/profile init scripts. It no
  longer issues a warning. This brings ksh93 in line with mksh, bash and zsh.

- A long-form shell option '-o posix' has been added, which implements a
  mode for better compatibility with the POSIX standard. It is automatically
  turned on if ksh is invoked under the name 'sh'.
  For now, it:
  * disables the &> redirection shorthand
  * causes the 'let' arithmetic command to recognise octal numbers by
    leading zeros regardless of the setting of the 'letoctal' option
  * causes file descriptors > 2 to be left open when invoking another program
  * makes the <> redirection operator default to stdin instead of stdout
    (this keeps the 2020-05-13 BUG_REDIRIO fix for the POSIX mode while
    restoring traditional ksh93 behaviour for backwards compatibility)
  * disables a noncompliant 'test -t' == 'test -t 1' compatibility hack
  * disables passing an exported variable's attributes (such as integer or
    readonly) to a new ksh process through the environment

2020-08-19:

- Sped up the 'read' command on most systems by 15-25%. Fixed a hanging bug
  on reading from a FIFO that could occur on macOS.

2020-08-17:

- 'command -p' incorrectly used the hash table entry (a.k.a. tracked alias)
  for a command if its path was previously hashed. It has now been fixed so
  it never consults the hash table.

2020-08-16:

- Fixed 'command -x' on macOS, Linux and Solaris by accounting for a 16-byte
  argument alignment. If execution does fail, it now aborts with an internal
  error message instead of entering an infinite retry loop.

2020-08-13:

- Fixed memory leaks and a crashing bug that occurred when defining and
  running functions in subshells.

2020-08-11:

- Fixed an intermittent crash upon running a large number of subshells.

2020-08-10:

- A number of fixes have been applied to the printf formatting directives
  %H and %#H (as well as the undocumented equivalents %(html)q and %(url)q):
  1. Both formatters have been made multibyte/UTF-8 aware, and no longer
     delete multibyte characters. Invalid UTF-8 byte sequences are rendered
     as ASCII question marks.
  2. %H no longer wrongly changes spaces to non-breaking spaces (&nbsp;).
  3. %H now converts the single quote (') to '%#39;' instead of '&apos;'
     which is not a valid entity in all HTML versions.
  4. %#H failed to encode some reserved characters (e.g. '?') while encoding
     some unreserved ones (e.g. '~'). It now percent-encodes all characters
     except those 'unreserved' as per RFC3986 (ASCII alphanumeric plus -._~).

- Fixed a crash that occurred intermittently after running an external
  command from a command substitution expanded from the $PS1 shell prompt.

2020-08-09:

- File name generation (a.k.a. pathname expansion, a.k.a. globbing) now
  never matches the special navigational names '.' (current directory) and
  '..' (parent directory). This change makes a pattern like .* useful; it
  now matches all hidden files (dotfiles) in the current directory, without
  the harmful inclusion of '.' and '..'.

2020-08-08:

- Argument checking in the 'redirect' builtin command (see 2020-06-11) has
  been improved to error out before executing redirections. For example, an
  error like 'redirect ls >foo.txt' now will not create 'foo.txt' and will
  not leave your standard output permanently redirected to it.

2020-08-06:

- Added the '${.sh.pid}' variable as an alternative to Bash's '$BASHPID'.
  This variable is set to the current shell's PID, unlike '$$' (which is
  set to the parent shell's PID). In virtual subshells '${.sh.pid}' is not
  changed from its previous value, while in forked subshells '${.sh.pid}'
  is set to the subshell's process ID.

2020-08-05:

- Fixed a bug in functions that caused ksh to crash when an array with an
  unset method was turned into a multidimensional array.

- Fixed a bug that caused scripts to continue running after over-shifting
  in a function when the function call had a redirection.

- When generating shellquoted strings (such as with 'printf %q'), the
  hexadecimal value of a quoted unprintable character was not protected with
  square braces, e.g. 0x12 followed by '3' would be quoted as '\x123', which
  is a different value. Such strings are now quoted like '\x[12]3' if the
  next character is a hexadecimal digit.

2020-07-31:

- Fixed a bug that caused multidimensional associative arrays to be created
  with an extra array member.

- Fixed a bug that caused the expansions of positional parameters $1 - $9,
  as well as special parameters such as $? and $-, to corrupt any multibyte
  characters immediately following the expansion if a UTF-8 locale is active.

2020-07-29:

- On a ksh compiled to use fork(2) to run external commands, a bug has been
  fixed that caused signals (such as SIGINT, Ctrl+C) to be ignored within a
  non-forked subshell after running an external command within that subshell.

2020-07-25:

- Fixed BUG_MULTIBIFS: Multibyte characters can now be used as IFS
  delimiters. "$*" was incorrectly joining positional parameters on
  the first byte of a multibyte character. This was due to truncation
  based on the incorrect assumption the IFS would never be larger
  than a single byte.

- Fixed a bug that caused the sleep builtin to continue after being given
  an unrecognized option. 'sleep -: 1' will now show a usage message and
  exit instead of sleep for one second.

- Fixed a bug that caused the 'typeset' variable attributes -a, -A, -l, and
  -u to leak out of a subshell if they were set without assigning a value.

2020-07-23:

- Fixed an infinite loop that could occur when ksh is the system's /bin/sh.

- A command substitution that is run on the same line as a here-document
  will no longer cause a syntax error.

2020-07-22:

- Fixed two race conditions when running external commands on
  interactive shells with job control active.

2020-07-20:

- If a shell function and a built-in command by the same name exist,
  'whence -a' and 'type -a' now report both.

- Fixed a bug that caused file descriptors opened with 'redirect' or 'exec'
  to survive a subshell environment after exiting it.

2020-07-19:

- Fixed a crash that occurred in the '.' command when using kshdb.

- Fixed a crash that occurred when attempting to use redirection with an
  invalid file descriptor.

2020-07-16:

- The 'history' and 'r' default aliases have been made regular built-ins,
  leaving zero default aliases.

- Fixed a bug that caused 'sleep -s' to have no effect with intervals longer
  than 30 seconds.

- The accuracy of the sleep builtin has been improved. It no longer ignores
  microseconds and doesn't add extra milliseconds when the interval is less
  than 31 seconds.

2020-07-15:

- The 'autoload', 'compound', 'float', 'functions', 'integer' and 'nameref'
  default aliases have been converted into regular built-in commands, so
  that 'unalias -a' does not remove them. Shell functions can now use
  these names, which improves compatibility with POSIX shell scripts.

- The End key escape sequence '^[[F' is now handled in the emacs and vi editing
  modes. The End key moves the cursor to the end of the line (in contrast to
  the Home key doing the opposite).

2020-07-14:

- Fixed a bug that caused 'set -b' to have no effect.

- Following the 'time' keyword, the 'times' builtin command now also
  supports millisecond precision.

2020-07-13:

- Fixed a fork bomb that could occur when the vi editor was sent SIGTSTP
  while running in a ksh script.

- Appending a lone percent to the end of a format specifier no longer
  causes a syntax error. The extra percent will be treated as a literal
  '%', like in Bash and zsh.

- The 'time' keyword now has proper support for millisecond precision.
  Although this feature was previously documented, the 'time' keyword
  only supported up to centisecond precision, which caused a command
  like the one below to return '0.000' on certain operating systems:
  $ TIMEFORMAT='%3R'; time sleep .003

- The 'time' keyword now zero-pads seconds less than ten (like mksh).

2020-07-10:

- Fixed a bug that caused types created with 'typeset -T' to throw an error
  when used if the type name started with a lowercase 'a'.

- A potential crash due to memory corruption when using many file
  descriptors has been fixed.

2020-07-09:

- Fixed a crash on syntax error when sourcing/dotting multiple files.

- Fixed a crash when listing indexed arrays.

- Fixed a memory leak when restoring PATH when temporarily setting PATH
  for a command (e.g. PATH=/foo/bar command ...) or in a virtual subshell.

- Combining ((...)) with redirections no longer causes a syntax error
  due to the parser handling '>' incorrectly.

- Fixed a bug that corrupted KIA/CQL cross-reference databases created using
  ksh's -R option; shell warnings were wrongly included in the database file.

- The shell's quoting algorithm (used in xtrace, printf %q, and more) has been
  fixed for UTF-8 (Unicode) locales; it no longer needlessly and inconsistently
  encodes normal printable UTF-8 characters into hexadecimal \u[xxxx] codes.

2020-07-07:

- Four of the date formats accepted by 'printf %()T' have had their
  functionality altered to the common behavior of date(1):
  - '%k' and '%l' print the current hour with blank padding, the former
    based on a 24-hour clock and the latter a twelve hour clock. These
    are common extensions present on Linux and *BSD.
  - '%f' prints a date with the format string '%Y.%m.%d-%H:%M:%S' (BusyBox).
  - '%q' prints the quarter of the year (GNU).

2020-07-06:

- 'notty' is now written to the ksh auditing file instead of '(null)' if
  the user's tty could not be determined.

- Unsetting an associative array no longer causes a memory leak to occur.

2020-07-05:

- In UTF-8 locales, fix corruption of the shell's internal string quoting
  algorithm (as used by xtrace, 'printf %q', and more) that occurred when
  the processing of a multibyte character was interrupted.

2020-07-03:

- Backslashes are no longer escaped in the raw Bourne Shell-like editing
  mode in multibyte locales, i.e. backslashes are no longer treated like
  Control-V if the emacs and vi modes are disabled.

- Deleting a backslash in vi mode with Control-H or Backspace now only
  escapes a backslash if it was the previous input. This means erasing a
  string such as 'ab\\\' will only cause the first backslash to escape a
  Backspace as '^?', like in emacs mode.

- An odd interaction with Backspace when the last character of a separate
  buffer created with Shift-C was '\' has been fixed. '^?' will no longer
  be output repeatedly when attempting to erase a separate buffer with
  a Backspace. Note that buffers created with Shift-C are not meant to be
  erasable:
  https://pubs.opengroup.org/onlinepubs/9699919799/utilities/vi.html#tag_20_152_13_49

- The 'kill' builtin now supports the SIGINFO signal (on operating systems
  with support for SIGINFO).

2020-07-02:

- Fixed a crash that occurred if a directory named '.paths' existed in any
  directory listed in $PATH. The fix was to only read '.paths' if it is a
  regular file or a symlink to a regular file.

2020-06-30:

- 'read -u' will no longer crash with a memory fault when given an out of
  range or negative file descriptor.

- The '=~' operator no longer raises an error if a regular expression
  combines the '{x}' quantifier with a sub-expression.

2020-06-28:

- Variables created with 'typeset -RF' no longer cause a memory fault
  when accessed.

- Unsetting an array that was turned into a compound variable will no
  longer cause silent memory corruption.

- Variables created with 'readonly' in functions are now set to the
  specified value instead of nothing. Note that 'readonly' does not
  create a function-local scope, unlike 'typeset -r' which does.

2020-06-26:

- Changing to a directory that has a name starting with a '.' will no
  longer fail if preceded by '../' (i.e. 'cd ../.local' will now work).

2020-06-24:

- Fixed buggy tab completion of tilde-expanded paths such as
  ~/some in 'vi' mode.

- In the raw/default Bourne Shell-like editing mode that occurs when neither
  the 'emacs' nor the 'vi' shell option is active:
  * tab completion is now correctly disabled, instead of enabled and broken;
  * entering tab characters now moves the cursor the correct amount.

2020-06-23:

- Fixed a bug that caused combining process substitution with redirection
  to create a bizarre file in the user's current working directory.

- Using process substitution while the shell is interactive no longer
  causes the process ID of the asynchronous process to be printed.

2020-06-22:

- The 'stop' and 'suspend' default aliases have been converted into regular
  built-in commands, so that 'unalias -a' does not remove them, 'suspend'
  can do a couple of sanity checks, and something like
	cmd=stop; $cmd $!
  will now work. See 'stop --man' and 'suspend --man' for more information.

- Fixed a bug that caused the kill and stop commands to segfault when given
  a non-existent job.

- Nested functions no longer ignore variable assignments that were prefixed
  to their parent function, i.e. 'VAR=foo func' will now set $VAR to 'foo'
  in the scope of any nested function 'func' runs.

2020-06-20:

- Fixed a bug that caused setting the following variables as readonly in
  a virtual subshell to affect the environment outside of the subshell:
  $_
  ${.sh.name}
  ${.sh.subscript}
  ${.sh.level}
  $RANDOM
  $LINENO

- Fixed two bugs that caused `unset .sh.lineno` to always produce a memory
  fault and `(unset .sh.level)` to memory fault when run in nested
  functions.

2020-06-18:

- A two decade old bug that caused 'whence -a' to base the path of
  tracked aliases on the user's current working directory has been
  fixed. Now the real path to tracked aliases is shown when '-a' is
  passed to the whence command.

2020-06-17:

- A bug in 'unset -f' was fixed that prevented shell functions from
  unsetting themselves while they were running. A POSIX function no longer
  crashes when doing so, and a KornShell-style function no longer silently
  ignores an 'unset -f' on itself. A function of either form now continues
  running after unsetting itself, and is removed at the end of the run.

2020-06-16:

- Passing the '-d' flag to the read builtin will no longer cause the '-r'
  flag to be discarded when 'read -r -d' is run.

- Fix BUG_CMDSPASGN: preceding a "special builtin"[*] with 'command' now
  prevents preceding invocation-local variable assignments from becoming global.
  [*] https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_14

2020-06-15:

- The 'source' alias has been converted into a regular built-in command.

- Functions that set variables in a virtual subshell will no longer affect
  variables of the same name outside of the virtual subshell's environment.

- Terse usage messages written by builtin commands now point the user to
  the --help and --man options for more information.

2020-06-14:

- 'read -S' is now able to correctly handle strings with double quotes
  nested inside of double quotes.

2020-06-13:

- Fixed a timezone name determination bug on FreeBSD that caused the
  output from `LC_ALL=C printf '%T' now` to print the wrong time zone name.

2020-06-11:

- Fixed a bug that caused running 'builtin -d' on a special builtin to
  delete it. The man page for the 'builtin' command documents that special
  builtins cannot be deleted.

- POSIX compliance fix: It is now possible to set shell functions named
  'alias' or 'unalias', overriding the commands by the same names. In
  technical terms, they are now regular builtins, not special builtins.

- The redirect='command exec' alias has been converted to a regular
  'redirect' builtin command that only accepts I/O redirections, which
  persist as in 'exec'. This means that:
  * 'unlias -a' no longer removes the 'redirect' command;
  * users no longer accidentally get logged out of their shells if
    they type something intuitive but wrong, like 'redirect ls >file'.

- The undocumented 'login' and 'newgrp' builtin commands have been removed.
  These replaced your shell session with the external commands by the same
  name, as in 'exec'. If an error occurred (e.g. due to a typo), you would
  end up immediately logged out.
  If you do want this behaviour, you can restore it by setting:
	alias login='exec login'
	alias newgrp='exec newgrp'

2020-06-10:

- The 'hash' utility is now a regular builtin instead of an alias to
  'alias -t --'. The functionality of the old command has been removed
  from the alias builtin.

- Changing the hash table in a subshell will no longer affect the parent
  shell's hash table. This fix applies to the hash utility and when the
  PATH is reset manually.

- 'set +r' is no longer able to unset the restricted option. This change
  makes the behavior of 'set +r' identical to 'set +o restricted'.

2020-06-09:

- The 'unalias' builtin will now return a non-zero status if it tries
  to remove a previously set alias that is not currently set.

2020-06-08:

- Fix an issue with the up arrow key in Emacs editing mode.
  Emacs editing mode is bugged in ksh93u+ and ksh2020. Let's
  say you were to run the following commands after starting
  a fresh instance of ksh:
  $ alias foo='true'
  $ unalias foo
  If you type 'a' and then press the up arrow on your keyboard,
  ksh will complete 'a' to `alias foo='true'` by doing a reverse
  search for the last command that starts with 'a'.
  Run the alias command again, then type 'u' and press the up
  arrow key again. If ksh is in Vi mode, you will get `unalias foo`,
  but in Emacs mode you will get `alias foo='true'` again.
  All subsequent commands were ignored as ksh was saving the first
  command and only based later searches off of it.

- If 'set -u'/'set -o nounset' is active, then the shell now errors out if a
  nonexistent positional parameter such as $1, $2, ... is accessed, as other
  shells do and POSIX requires. (This does *not* apply to "$@" and "$*".)

- If 'set -u'/'set -o nounset' is active, then the shell now errors out if $!
  is accessed before the shell has launched any background process.

- Removed support for an obscure early 1990s Bell Labs file system research
  project called 3DFS, which has not existed for decades. This removes:
  - an obnoxious default alias 2d='set -f;_2d' that turned off your file name
    wildcard expansion and then tried to run a nonexistent '_2d' command
  - undocumented builtins 'vmap' and 'vpath' that only printed error messages
  - a non-functional -V unary operator for the test and [[ commands

- If the last program run by a ksh script exits with a signal (e.g. crashed),
  ksh itself now exits normally instead of repeating that same signal.
  In addition, using 'exit x' for x > 256 no longer makes ksh issue a signal.

2020-06-06:

- The 'times' command is now a builtin command that conforms to POSIX
  instead of an alias for the 'time' command. It displays the accumulated
  user and system CPU times, one line with the times used by the shell and
  another with those used by all of the shell's child processes.
  https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_27

- The default aliases command='command ' and nohup='nohup ' have been
  removed because they caused breakage in an attempt to circumvent other
  breakage which is being fixed. In the unlikely even that anyone still
  needs alias substitution to continue on the command argument following
  'command' or 'nohup', it's easy to set these aliases yourself.

2020-06-05:

- Fix a bug that caused special variables such as PATH, LANG, LC_ALL,
  etc. to lose their effect after being unset in a subshell. For example:
    (unset PATH; PATH=/dev/null; ls); : wrongly ran 'ls'
    (unset LC_ALL; LC_ALL=badlocale); : failed to print a diagnostic
  This also fixes BUG_KUNSETIFS: unsetting IFS in a subshell failed if IFS
  was set to the empty value in the parent shell.

- Fix crashes on some systems, including at least a crash in 'print -v' on
  macOS, by eliminating an invalid/undefined use of memccpy() on overlapping
  buffers in the commonly used sfputr() function.

- Fix the ${.sh.subshell} level counter; it is no longer reset to zero when a
  non-forked subshell happens to fork into a separate process for some reason
  (an internal implementation detail that should be unnoticeable to scripts).

2020-06-04:

- Fix BUG_KBGPID: the $! special parameter was not set if a background job
  (somecommand &) or co-process (somecommand |&) was launched as the only
  command within a braces block with an attached redirection, for example:
	{
		somecommand &
	} >&2
  With the bug, $! was unchanged; now it contains the PID of somecommand.

2020-05-31:

- Fix a bug in autoloading functions. Directories in the path search list
  which should be skipped (e.g. because they don't exist) did not interact
  correctly with autoloaded functions, so that a function to autoload was
  not always found correctly.
  Details: https://github.com/att/ast/issues/1454

2020-05-30:

- Fix POSIX compliance of 'test'/'[' exit status on error. The command now
  returns status 2 instead of 1 when given an invalid number or arithmetic
  expression, e.g.:
	[ 123 -eq 123x ]; echo $?
  now outputs 2 instead of 1.

2020-05-29:

- Fix BUG_FNSUBSH: functions can now be correctly redefined and unset in
  subshell environments (such as ( ... ), $(command substitutions), etc).
  Before this fix, this was silently ignored, causing the function by the
  same name from the parent shell environment to be executed instead.
	fn() { echo mainsh; }
	(fn() { echo subsh; }; fn); fn
  This now correctly outputs "subsh mainsh" instead of "mainsh mainsh".
	ls() { echo "ls executed"; }
	(unset -f ls; ls); ls
  This now correctly lists your directory and then prints "ls executed",
  instead of printing "ls executed" twice.

- Fix a similar bug with aliases. These can now be correctly unset
  in subshell environments.

2020-05-21:

- Fix truncating of files with the combined redirections '<>;file' and
  '<#pattern'. The bug was caused by out-of-sync streams.
  Details and discussion: https://github.com/att/ast/issues/61

- Patched code injection vulnerability CVE-2019-14868. As a result, you can
  no longer use expressions in imported numeric environment variables; only
  integer literals are allowed.

2020-05-20:

- Fix BUG_ISSETLOOP. Expansions like ${var+set} remained static when used
  within a 'for', 'while' or 'until' loop; the expansions din't change along
  with the state of the variable, so they could not be used to check whether a
  variable is set within a loop if the state of that variable changed in the
  course of the loop.

- Fix BUG_IFSISSET. ${IFS+s} always yielded 's', and [[ -v IFS ]] always
  yielded true, even if IFS is unset. This applied to IFS only.

2020-05-19:

- Fix 'command -p'. The -p option causes the operating system's standard
  utilities path (as output by 'getconf PATH') to be searched instead of $PATH.
  Before this fix, this was broken on non-interactive shells as the internal
  variable holding the default PATH value was not correctly initialised.

2020-05-16:

- Fix 'test -t 1', '[ -t 1 ]', '[[ -t 1 ]]' in command substitutions.
  Standard output (file descriptor 1) tested as being on a terminal within a
  command substitution, which makes no sense as the command substitution is
  supposed to be catching standard output.
	v=$(echo begincomsub
	    [ -t 1 ] && echo oops
	    echo endcomsub)
	echo "$v"
  This now does not output "oops".

2020-05-14:

- Fix syncing history when print -s -f is used. For example, the
  following now correctly adds a 'cd' command to the history:
	print -s -f 'cd -- %q\n' "$PWD"
  Ref.:	https://github.com/att/ast/issues/425
	https://github.com/att/ast/pull/442

- Fix BUG_PUTIOERR: Output builtins now correctly detect
  input/output errors. This allows scripts to check for a nonzero exit
  status on the 'print', 'printf' and 'echo' builtins and prevent possible
  infinite loops if SIGPIPE is ignored.

- Add a convenient bin/run_ksh_tests script to the source tree that
  sets up the necessary environment and runs the ksh regression tests.

2020-05-13:

- Fix BUG_CASELIT: an undocumented 'case' pattern matching misbehaviour that
  goes back to the original Bourne shell, but wasn't discovered until 2018.
  If a pattern doesn't match as a pattern, it was tried again as a literal
  string. This broke common validation use cases, e.g.:
      n='[0-9]'
      case $n in
      ( [0-9] )  echo "$n is a number" ;;
      esac
  would output "[0-9] is a number" as the literal string fallback matches the
  pattern. As this misbehaviour was never documented anywhere (not for Bourne,
  ksh88, or ksh93), and it was never replicated in other shells (not even in
  ksh88 clones pdksh and mksh), it is unlikely any scripts rely on it.
     Of course, a literal string fallback, should it be needed, is trivial to
  implement correctly without this breakage:
      case $n in
      ( [0-9] | "[0-9]")  echo "$n is a number or the number pattern" ;;
      esac
  Ref.: https://github.com/att/ast/issues/476

- Fix BUG_REDIRIO: ksh used to redirect standard output by default when no
  file descriptor was specified with the rarely used '<>' reading/writing
  redirection operator. It now redirects standard input by default, as POSIX
  specifies and as all other POSIX shells do. To redirect standard output
  for reading and writing, you now need '1<>'.
  Ref.:	https://github.com/att/ast/issues/75
	http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_07_07