1
0
Fork 0
mirror of git://git.code.sf.net/p/cdesktopenv/code synced 2025-03-09 15:50:02 +00:00

tests/leaks.sh: redesign with a more robust testing algorithm

On modern operating systems, memory management is non-deterministic
(i.e. random, unpredictable) to varying degrees. This makes testing
for memory leaks a nightmare as the OS may decide to randomly grow
a process's memory allocation at any time for no apparent reason,
causing intermittent test failures that do not represent real
memory leaks. So far, the leaks test tried to cope with this by
using a large number of iterations plus a certain amount of bytes
of tolerance per iteration. This was inefficient and on some
systems still did not fully eliminate intermittent test failures.

This commit introduces a new testing algorithm that is designed to
cope with a large degree of unpredictability. Instead of a fixed
number of test iterations, it defines a maximum (16384), dividing
them in blocks of 128 iterations. It also defines a minimum number
of sequential "good" iteration blocks, counted if memory usage did
not increase from one block to the next. That minimum number is set
to 16. The theory is that if we can get 16 "good" iteration blocks
in a row, we can safely assume it's not a real memory leak, break
the loop, and consider the test succeeded. That "good" sequence is
allowed to occur at any point in the loop, creating a high built-in
tolerance for non-deterministic shenanigans. It also speeds up the
tests, as successful tests can bow out at 16 * 128 == 2048
iterations if they're lucky. If the OS decides to randomly grow the
memory heap, it may take more tries, but almost (?) certainly not
more than the maximum 16384 (128 blocks). If the counter reaches
that, then we assume a memory leak and throw a test failure.

We're also no longer testing with byte granularity in any case; the
randomness of memory management makes that pointless. All getmem()
function versions now return kibibytes (1024 bytes).

This should eliminate the need for workarounds such as initial
iterations to "steady the state" or a tolerance of a certain number
of bytes. I've experimentally determined the exact values
(max_iter, block_iter, min_good_blocks) that seem to work reliably
on all systems I've tested. They are easy to tweak if necessary.

To make all this manageable, this commit hides all the supporting
code in a triplet of aliases (TEST, DO, DONE) that, when used
correctly, create a grammatically robust shell code block: you can
add redirections, pipe into it, etc. as expected. This makes the
actual tests a great deal easier to read as well.

src/cmd/ksh93/tests/pty.sh:
- Implement new leaks testing framework as described and convert
  all the tests to it.
- Mark known leaks with a 'known' variable. Print non-fail warnings
  for all known leaks, but skip the tests by default. Test them
  only if DEBUG is exported. This is better than commenting them
  out as we will no longer be tempted to forget about these.
- Move the test for large command substitutions to subshell.sh --
  it's not in fact a leak test; instead, it checks that command
  substitutions don't lose data.

src/cmd/ksh93/tests/_common: err_exit():
- Since we're printing more warnings, clearly mark all test
  failures with 'FAIL:' to make them stand out.

src/cmd/ksh93/tests/shtests.
src/cmd/ksh93/tests/pty.sh:
- Special-case leaks.sh for counting tests; grep ^TEST.
- Special-case pty.sh as well while we're at it by grepping tst()
  calls. Remove all the dummy '# err_exit #' comments from pty.sh
  as they are now no longer used for counting the tests.
This commit is contained in:
Martijn Dekker 2021-12-28 17:47:29 +00:00
parent a9c6f77c3e
commit db3a3d8fc0
5 changed files with 259 additions and 369 deletions

View file

@ -8,7 +8,7 @@ valgrindflags='--xml=yes --log-file=/dev/null --track-origins=yes --read-var-inf
USAGE=$'
[-s8?
@(#)$Id: shtests (ksh 93u+m) 2021-12-20 $
@(#)$Id: shtests (ksh 93u+m) 2021-12-28 $
]
[-author?David Korn <dgk@research.att.com>]
[-author?Glenn Fowler <gsf@research.att.com>]
@ -344,6 +344,8 @@ do [[ $i == *.sh ]] || i+='.sh'
fi
t=$( case $i in
glob.sh) grep -c '^[[:blank:]]*test_[a-z]\{3,\}' $i ;;
leaks.sh) grep -c ^TEST $i ;;
pty.sh) grep -c 'tst ' $i ;;
*) grep -c err_exit $i ;;
esac )
tests[$i]=$t