external/cde
1
0
Fork 0
mirror of git://git.code.sf.net/p/cdesktopenv/code synced 2025-03-09 15:50:02 +00:00
Code Issues Releases Wiki Activity
No description
403 commits 3 branches 25 tags 132 MiB
Find a file
Martijn Dekker bb15f7fb19 Fix elusive/intermittent DEBUG trap crash (rhbz#1200534) 
For one Red Hat customer, the following reproducer consistently
crashed, tough I was not able to reproduce it and neither was RH.
However, the crash analysis is sound (see below).

    function dlog
    {
      fc -ln -0
    }
    trap dlog DEBUG
    >/tmp/blah

Original patch: 642af4d6/f/ksh-20140801-arraylen.patch

The Red Hat bug thread is closed to the public as it also contains
some correspondence with their customer. But it has an excellent
crash analysis from Thomas Gardner which I'm including here for the
record (the line numbers are for their ksh at the time, not 93u+m).

===begin analysis===
> The creation of an empty file instead of a command that executes
> anything causes the coredump.
[...]
> Here is my analysis on the core that was provided by the customer:
>
> (gdb) bt
> #0  sh_fmtq (string=0x1 <Address 0x1 out of bounds>)
>     at /usr/src/debug/ksh-20120801/src/cmd/ksh93/sh/string.c:340
> #1  0x0000000000457e96 in out_string (cp=<value optimized out>, c=32,
>     quoted=<value optimized out>, iop=<value optimized out>)
>     at /usr/src/debug/ksh-20120801/src/cmd/ksh93/sh/xec.c:444
> #2  0x000000000045804c in sh_debug (shp=0x76d180, trap=0x7f2f13a821e0 "dlog",
>     name=<value optimized out>, subscript=<value optimized out>,
>     argv=0x76e070, flags=<value optimized out>)
>     at /usr/src/debug/ksh-20120801/src/cmd/ksh93/sh/xec.c:548
> #3  0x000000000045a867 in sh_exec (t=0x7f2f13aafad0, flags=4)
>     at /usr/src/debug/ksh-20120801/src/cmd/ksh93/sh/xec.c:1265
> [...need go no further...]
>
> In frame 2, we can see it cycling through your classic
> (char **)argv array like:
>
> 543		while(cp = *argv++)
> 544		{
> 545			if((flags&ARG_EXP) && argv[1]==0)
> 546				out_pattern(iop, cp,' ');
> 547			else
> 548				out_string(iop, cp,' ',n?0: (flags&(ARG_RAW|ARG_NOGLOB))||*argv);
> 549		}
> 550		if(flags&ARG_ASSIGN)
> 551			sfputc(iop,')');
> 552		else if(iop==stkstd)
>
> (we seg-fault after going down the out_string function in line
> 548 up there). The string pointer that points to = 0x1 up in
> frame #0 (sh_fmtq) traces back to the "cp" variable in line 548
> up there. The "argv" variable being referenced up there just gets
> passed in as the fifth argument to this function.
>
> In frame #3 (sh_exec, line 1265), the line that makes the call
> that takes us to frame 2 is:
>
> 1265                     int n = sh_debug(shp,trap,(char*)0,(char*)0, com, ARG_R     AW);
>
> so "com" (the fifth argument) is what's going wrong as it
> descends down through these calls. Looking at where it comes
> from, well, it's assigned here:
>
> 1241                 if(argn==0)
> 1242                 {
> 1243                     /* fake 'true' built-in */
> 1244                     np = SYSTRUE;
> 1245                     *argv = nv_name(np);
> 1246                     com = argv;
> 1247                 }
>
> because as we can see:
>
> (gdb) f 3
> #3  0x000000000045a867 in sh_exec (t=0x7f2f13aafad0, flags=4)
>     at /usr/src/debug/ksh-20120801/src/cmd/ksh93/sh/xec.c:1265
> 1265						int n = sh_debug(shp,trap,(char*)0,(char*)0, com, ARG_RAW);
> (gdb) p argn
> $2 = 0
> (gdb)
>
> argn is == 0 here. The tip-off here is that nv_name clearly
> returns a simple pointer to an array of characters, not an array
> of pointers to arrays of characters as is evidenced by the fact
> that the assignment is "*argv = nv_name(np);" not "argv =
> nv_name(np);". Looking at the function nv_name proves that it
> does indeed return a single pointer to an array of characters,
> not a pointer to an array of pointers to arrays of characters.
> Now, com is defined as a 'char **':
>
> 1002         char        *cp=0, **com=0, *comn;
>
> (as it is expected to be in the calls that follow) also, that
> argv is also defined as the effective equivalent a 'char **':
>
> 1237                 static char *argv[1];
>
> Yup, argv is actually an array of pointers (char ** equivalent),
> but that array is restricted to having exactly one element.
> Recalling the assignment in the previously quoted line:
>
> 1245                     *argv = nv_name(np);
>
> we see that the one and only element in that argv array is
> getting assigned a pointer to an array of characters here.
> Nothing necessarily wrong with that, but remember the loop we
> looked at earlier in frame #2 (sh_debug). It went like:
>
> 543		while(cp = *argv++)
> 544		{
> 545			if((flags&ARG_EXP) && argv[1]==0)
> 546				out_pattern(iop, cp,' ');
> 547			else
> 548				out_string(iop, cp,' ',n?0: (flags&(ARG_RAW|ARG_NOGLOB))||*argv);
> 549		}
>
> which is clearly expecting argv in this context (com in frame 3,
> which really points to that static local single element array
> that is also pointed to by argv in frame 2) to be an array of
> pointers of indefinite size, each element being a pointer, but
> whose last element will be a null pointer. Well, in frame 3 it is
> clearly an array with only a single element, and that one element
> is NOT pointing to null. Watch this:
>
> (gdb) f 3
> #3  0x000000000045a867 in sh_exec (t=0x7f2f13aafad0, flags=4)
>     at /usr/src/debug/ksh-20120801/src/cmd/ksh93/sh/xec.c:1265
> 1265						int n = sh_debug(shp,trap,(char*)0,(char*)0, com, ARG_RAW);
> (gdb) p com
> $8 = (char **) 0x76e060
> (gdb) p &argv
> $9 = (char *(*)[1]) 0x76e060
> (gdb) p com[0]
> $11 = 0x5009c6 "true"
> (gdb) p com[1]
> $10 = 0x1 <Address 0x1 out of bounds>
> (gdb) p argv[0]
> $12 = 0x5009c6 "true"
> (gdb) p argv[1]
> $13 = 0x1 <Address 0x1 out of bounds>
> (gdb)
>
> So, as expected, com and &argv point to the same place, the first
> element points to the constant string "true", but since the array
> is defined as having only one element, when you refer to a second
> element in that array, you get well, whatever random crap happens
> to be in that memory location. When we try to reproduce this
> problem, apparently we're getting 0 there (or we're not quite
> following this same code path, which is also possible), but the
> customer happens to have a "1" in that memory location.
===end analysis===

src/cmd/ksh93/sh/xec.c: sh_exec():
- When processing TCOM (simple command) with an empty/null command,
  increase the size of the static dummy argv[1] array to argv[2],
  ensuring a terminating NULL element so that 'while(cp = *argv++)'
  loops don't crash. (Note that static objects are automatically
  initialised to zero in C.)

src/cmd/ksh93/tests/io.sh:
- Adapt the reproducer, testing a null-command redirection 1000x.
2020-09-27 13:20:03 +02:00
.github/workflows tests/builtins.sh: change GMT to UTC before testing (re: c9634e90) 2020-06-26 13:42:06 +02:00
bin bin/package: don't test-compile using possibly broken dev shell 2020-09-22 16:14:53 +02:00
docs Fix a large number of typos and other problems (#110) 2020-08-07 00:50:11 +01:00
lib/package Fix a large number of typos and other problems (#110) 2020-08-07 00:50:11 +01:00
src Fix elusive/intermittent DEBUG trap crash (rhbz#1200534) 2020-09-27 13:20:03 +02:00
.gitignore Fix spurious creation of '=' file (#98) 2020-07-27 13:27:20 +01:00
LICENSE.md Create LICENSE.md 2020-02-12 14:03:19 -05:00
NEWS Avoid importing env vars with invalid names (rhbz#1147645) 2020-09-26 20:57:39 +02:00
README.md Continous integration builds using Github actions (#36) 2020-06-22 23:53:26 +01:00
TODO Fix 'command' expansion bug and POSIX compliance 2020-09-11 10:06:43 +02:00

README.md

KornShell 93u+m

This repository is used to develop bugfixes to the last stable release (93u+ 2012-08-01) of ksh93, formerly developed by AT&T Software Technology (AST). The sources in this repository were forked from the Github AST repository which is no longer under active development.

To see what's fixed, see NEWS and click on commit messages for full details.

To see what's left to fix, see TODO.

Policy

  1. No new features; bug fixes only (but see items 3 and 4). Feature development is for a future separate branch.
  2. No major rewrites. No refactoring code that is not fully understood.
  3. No changes in documented behaviour, except if required for compliance with the POSIX shell language standard which David Korn intended for ksh to follow.
  4. No 100% bug compatibility. Broken and undocumented behaviour gets fixed.
  5. No bureaucracy, no formalities. Just fix it, or report it: create issues, send pull requests. Every interested party is invited to contribute.
  6. To help increase everyone's understanding of this code base, fixes and significant changes should be fully documented in commit messages.

Why?

Between 2017 and 2020 there was an ultimately unsuccessful attempt to breathe new life into the KornShell by extensively refactoring the last unstable AST beta version (93v-). While that ksh2020 branch is now abandoned and still has many critical bugs, it also had a lot of bugs fixed. More importantly, the AST issue tracker now contains a lot of documentation on how to fix those bugs, which makes it possible to backport many of them to the last stable release instead.

In February 2020, having concluded the AST 93v- beta was too broken to base new work on, others decided to start a new fork based on the last stable 93u+ 2012-08-01 release. Unfortunately, as of June 2020, the new ksh-community organisation is yet to see any significant activity four months after its bootstrapping. I hope that will change; I am ready to join efforts with them at any time, as well as anyone else who wants to contribute.

The last stable ksh93 release from 2012 is the least buggy release currently available, but it still has many serious bugs. So it is well past time to start fixing those bugs, leave the rest of the code alone, and get an improved release out there.

Build

After cloning this repo, cd to the top directory of it and run:

bin/package make

If you have trouble or want to tune the binaries, you may pass additional compiler and linker flags by appending it to the command shown above. E.g.:

bin/package make \
    SHELL=/bin/bash CCFLAGS="-xc99 -D_XPG6 -m64 -xO4" LDFLAGS="-m64"

For more information run

bin/package help

Many other commands in this repo self-document via the --help, --man and --html options; those that do have no separate manual page.

Test

After compiling, you can run the regression tests. Start by reading the information printed by:

bin/shtests --man

What is ksh93?

The following is the official AT&T description from 1993 that came with the ast-open distribution. The text is original, but hyperlinks were added here.

KSH-93 is the most recent version of the KornShell Language described in "The KornShell Command and Programming Language," by Morris Bolsky and David Korn of AT&T Bell Laboratories, ISBN 0-13-182700-6. The KornShell is a shell programming language, which is upward compatible with "sh" (the Bourne Shell), and is intended to conform to the IEEE P1003.2/ISO 9945.2 Shell and Utilities standard. KSH-93 provides an enhanced programming environment in addition to the major command-entry features of the BSD shell "csh". With KSH-93, medium-sized programming tasks can be performed at shell-level without a significant loss in performance. In addition, "sh" scripts can be run on KSH-93 without modification.

The code should conform to the IEEE POSIX 1003.1 standard and to the proposed ANSI-C standard so that it should be portable to all such systems. Like the previous version, KSH-88, it is designed to accept eight bit character sets transparently, thereby making it internationally compatible. It can support multi-byte characters sets with some characteristics of the character set given at run time.

KSH-93 provides the following features, many of which were also inherent in KSH-88:

  • Enhanced Command Re-entry Capability: The KSH-93 history function records commands entered at any shell level and stores them, up to a user-specified limit, even after you log off. This allows you to re-enter long commands with a few keystrokes - even those commands you entered yesterday. The history file allows for eight bit characters in commands and supports essentially unlimited size histories.
  • In-line Editing: In "sh", the only way to fix mistyped commands is to backspace or retype the line. KSH-93 allows you to edit a command line using a choice of EMACS-TC or "vi" functions. You can use the in-line editors to complete filenames as you type them. You may also use this editing feature when entering command lines from your history file. A user can capture keystrokes and rebind keys to customize the editing interface.
  • Extended I/O Capabilities: KSH-93 provides several I/O capabilities not available in "sh", including the ability to:
    • specify a file descriptor for input and output
    • start up and run co-processes
    • produce a prompt at the terminal before a read
    • easily format and interpret responses to a menu
    • echo lines exactly as output without escape processing
    • format output using printf formats.
    • read and echo lines ending in "\".
  • Improved performance: KSH-93 executes many scripts faster than the System V Bourne shell. A major reason for this is that many of the standard utilities are built-in. To reduce the time to initiate a command, KSH-93 allows commands to be added as built-ins at run time on systems that support dynamic loading such as System V Release 4.
  • Arithmetic: KSH-93 allows you to do integer arithmetic in any base from two to sixty-four. You can also do double precision floating point arithmetic. Almost the complete set of C language operators are available with the same syntax and precedence. Arithmetic expressions can be used to as an argument expansion or as a separate command. In addition there is an arithmetic for command that works like the for statement in C.
  • Arrays: KSH-93 supports both indexed and associative arrays. The subscript for an indexed array is an arithmetic expression, whereas, the subscript for an associative array is a string.
  • Shell Functions and Aliases: Two mechanisms - functions and aliases - can be used to assign a user-selected identifier to an existing command or shell script. Functions allow local variables and provide scoping for exception handling. Functions can be searched for and loaded on first reference the way scripts are.
  • Substring Capabilities: KSH-93 allows you to create a substring of any given string either by specifying the starting offset and length, or by stripping off leading or trailing substrings during parameter substitution. You can also specify attributes, such as upper and lower case, field width, and justification to shell variables.
  • More pattern matching capabilities: KSH-93 allows you to specify extended regular expressions for file and string matches.
  • KSH-93 uses a hierarchal name space for variables. Compound variables can be defined and variables can be passed by reference. In addition, each variable can have one or more disciplines associated with it to intercept assignments and references.
  • Improved debugging: KSH-93 can generate line numbers on execution traces. Also, I/O redirections are now traced. There is a DEBUG trap that gets evaluated before each command so that errors can be localized.
  • Job Control: On systems that support job control, including System V Release 4, KSH-93 provides a job-control mechanism almost identical to that of the BSD "csh", version 4.1. This feature allows you to stop and restart programs, and to move programs between the foreground and the background.
  • Added security: KSH-93 can execute scripts which do not have read permission and scripts which have the setuid and/or setgid set when invoked by name, rather than as an argument to the shell. It is possible to log or control the execution of setuid and/or setgid scripts. The noclobber option prevents you from accidentally erasing a file by redirecting to an existing file.
  • KSH-93 can be extended by adding built-in commands at run time. In addition, KSH-93 can be used as a library that can be embedded into an application to allow scripting.

Documentation for KSH-93 consists of an "Introduction to KSH-93", "Compatibility with the Bourne Shell" and a manual page and a README file. In addition, the "New KornShell Command and Programming Language" book is available from Prentice Hall.