Improve TON build scripts and some tests (#855)

* fix macOS github actions

* fix android tonlib GH action;

* fixing wasm GH action

* strip binaries

* fix randomly failing ubuntu and wasm GH actions

* fix randomly failing ubuntu and wasm GH actions

* revert some changes

* adding more nix scripts and automated native build scripts;
debug static ton compilation

* minor fix

* do not use pkg_config if path specified

* move wasm script, run with sudo action script

* weird, fixing - cp: missing destination file operand after 'assembly/native/build-ubuntu-20.04-shared.sh'

* weird, fixing - cp: missing destination file operand after 'assembly/native/build-ubuntu-20.04-shared.sh'

* ok

* some adjustments for android and win builds

* some adjustments for android and win builds

* moving stripping inside the build script

* access rights handling; adding simple binaries' tests

* make lite-client-common, fift-lib and smc-envelope deliberately static;
add -a (artifacts) flag to build scripts;

* minor wasm build fix;
create separate tonlib android build script;
remove outdated __has_trivial_copy(T)

* add windows build - WIP

* adjust android build;
improve win build;

* adjust sodium paths for android build; use proper compiler for windows build;

* add github windows build auxiliary file

* adjust wasm build

* add portable ubuntu build

* exclude some unstable tests for some time

* compile portable binaries on ubuntu-20.04

* exclude some unstable tests

* include static gsl

* restart builds

* restart builds

* restart builds

* remove libreadline, gsl and blas dependencies in linux build

* add macos build script

* install missing autoconf in macos builds

* enable all tests and see what fails

* enable win tests and restart others

* enable win tests and fix test-smartcont.cpp

* enable win tests

* use clang-16 on mac builds, add blockchain-explorer for ubuntu builds, add portable macos build

* move sudo part outside a build scripts

* move sudo part outside a build scripts

* run llvm install with sudo

* remove libgnutls28-dev before ubuntu static compilation, include blockchain-explorer into artifacts;
remove warning: definition of implicit copy constructor for 'Stat' is deprecated because it has a user-declared copy assignment operator [-Wdeprecated-copy]

* rework wrong decision, put back system gnutls, but compile libmicrohttpd with --disable-https

* add jenkins pipeline sceleton

* WIP jenkins pipeline sceleton

* WIP jenkins pipeline changes

* WIP jenkins pipeline: add stage timout, zip and group artifacts

* WIP jenkins pipeline: macos portable build fix

* WIP jenkins pipeline: wording

* WIP jenkins pipeline: add android tonlib

* WIP jenkins pipeline: add was binaries

* WIP jenkins pipeline: add TOTAL_MEMORY 1.5gb to funcfiftlib wasm linking

* WIP jenkins pipeline: add nix build on linux aarch64

* WIP jenkins pipeline: funcfiftlib compilation fails that 16mb mem is not enough, increase to 32mb

* WIP jenkins pipeline: enable test in nix build

* WIP jenkins pipeline: add linux x86-64 nix build

* WIP jenkins pipeline: include libs in nix build

* WIP jenkins pipeline: include libs in nix build

* WIP jenkins pipeline: include mac nix build

* WIP jenkins pipeline: include mac nix build

* WIP jenkins pipeline: include mac nix build

* WIP jenkins pipeline: include mac nix build

* WIP jenkins pipeline: include mac nix build

* WIP jenkins pipeline: include mac nix build

* WIP jenkins pipeline: nix linux arm64 with openssl 1.1 for now

* WIP jenkins pipeline: working ubuntu arm64 libtonjson

* WIP jenkins pipeline: working ubuntu arm64 libtonjson + minor fix

* WIP jenkins pipeline: working ubuntu arm64 libtonjson + minor fix 2

* WIP jenkins pipeline: merry christmas

* WIP jenkins pipeline: merry christmas 2

* WIP jenkins pipeline: remove native static builds

* WIP jenkins pipeline: enable more tests

* WIP jenkins pipeline: zip artifacts better

* WIP jenkins pipeline: get rid of path in the final zip

* WIP jenkins pipeline: minor fix, include lib and smartcont folders

* WIP jenkins pipeline: minor fix, include lib and smartcont folders into nix artifacts also

* WIP jenkins pipeline: minor fix

* WIP jenkins pipeline: minor fix

* adjust github actions for new nix builds

* cleanup

* cleanup

* cleanup

* cleanup

* rename libtonlibjson.so.0.5 to libtonlibjson.so

* Add TON build instructions to README.md

* simplify

* fix test-tonlib-offline

* set timeout per test of 300 sec

* set timeout per test of 600 sec for non nix builds

* increase test timeout to 900 sec; minor changes

* use MS VS 2022 for win TON compilation; update README.md

* use MS VS 2022 for win TON compilation; update README.md

* change path to MSVC in github workflow

* change path to MSVC in groovy pipeline

* compile ton on win, with msvc 2022 community and enterprise versions

* minor fixes

* improve network tests

* remove TON compilation against macos-11 github runner

* add `choco feature enable -n allowEmptyChecksums` since pkg-config-lite-0.28-1 does not have a checksum

* abort win compilation if 3pp can't be downloaded

* increase test timeout to 30 min

* improving test-catchain

assembly/native/build-macos-portable.sh

@@ -0,0 +1,207 @@
+#/bin/bash
+
+with_tests=false
+with_artifacts=false
+OSX_TARGET=10.15
+
+
+while getopts 'tao:' flag; do
+  case "${flag}" in
+    t) with_tests=true ;;
+    a) with_artifacts=true ;;
+    o) OSX_TARGET=${OPTARG} ;;
+    *) break
+       ;;
+  esac
+done
+
+if [ ! -d "build" ]; then
+  mkdir build
+  cd build
+else
+  cd build
+  rm -rf .ninja* CMakeCache.txt
+fi
+
+export NONINTERACTIVE=1
+brew install ninja pkg-config automake libtool autoconf
+brew install llvm@16
+
+
+if [ -f /opt/homebrew/opt/llvm@16/bin/clang ]; then
+  export CC=/opt/homebrew/opt/llvm@16/bin/clang
+  export CXX=/opt/homebrew/opt/llvm@16/bin/clang++
+else
+  export CC=/usr/local/opt/llvm@16/bin/clang
+  export CXX=/usr/local/opt/llvm@16/bin/clang++
+fi
+export CCACHE_DISABLE=1
+
+if [ ! -d "secp256k1" ]; then
+git clone https://github.com/bitcoin-core/secp256k1.git
+cd secp256k1
+secp256k1Path=`pwd`
+git checkout v0.3.2
+./autogen.sh
+./configure --enable-module-recovery --enable-static --disable-tests --disable-benchmark --with-pic
+make -j12
+test $? -eq 0 || { echo "Can't compile secp256k1"; exit 1; }
+cd ..
+else
+  secp256k1Path=$(pwd)/secp256k1
+  echo "Using compiled secp256k1"
+fi
+
+if [ ! -d "libsodium" ]; then
+  export LIBSODIUM_FULL_BUILD=1
+  git clone https://github.com/jedisct1/libsodium.git
+  cd libsodium
+  sodiumPath=`pwd`
+  git checkout 1.0.18
+  ./autogen.sh
+  ./configure --with-pic --enable-static
+  make -j12
+  test $? -eq 0 || { echo "Can't compile libsodium"; exit 1; }
+  cd ..
+else
+  sodiumPath=$(pwd)/libsodium
+  echo "Using compiled libsodium"
+fi
+
+if [ ! -d "openssl_3" ]; then
+  git clone https://github.com/openssl/openssl openssl_3
+  cd openssl_3
+  opensslPath=`pwd`
+  git checkout openssl-3.1.4
+  ./config -static
+  make build_libs -j12
+  test $? -eq 0 || { echo "Can't compile openssl_3"; exit 1; }
+  cd ..
+else
+  opensslPath=$(pwd)/openssl_3
+  echo "Using compiled openssl_3"
+fi
+
+if [ ! -d "zlib" ]; then
+  git clone https://github.com/madler/zlib.git
+  cd zlib
+  zlibPath=`pwd`
+  ./configure --static
+  make -j12
+  test $? -eq 0 || { echo "Can't compile zlib"; exit 1; }
+  cd ..
+else
+  zlibPath=$(pwd)/zlib
+  echo "Using compiled zlib"
+fi
+
+if [ ! -d "libmicrohttpd" ]; then
+  git clone https://git.gnunet.org/libmicrohttpd.git
+  cd libmicrohttpd
+  libmicrohttpdPath=`pwd`
+  ./autogen.sh
+  ./configure --enable-static --disable-tests --disable-benchmark --disable-shared --disable-https --with-pic
+  make -j12
+  test $? -eq 0 || { echo "Can't compile libmicrohttpd"; exit 1; }
+  cd ..
+else
+  libmicrohttpdPath=$(pwd)/libmicrohttpd
+  echo "Using compiled libmicrohttpd"
+fi
+
+cmake -GNinja .. \
+-DPORTABLE=1 \
+-DCMAKE_OSX_DEPLOYMENT_TARGET:STRING=$OSX_TARGET \
+-DCMAKE_CXX_FLAGS="-stdlib=libc++" \
+-DCMAKE_BUILD_TYPE=Release \
+-DOPENSSL_FOUND=1 \
+-DOPENSSL_INCLUDE_DIR=$opensslPath/include \
+-DOPENSSL_CRYPTO_LIBRARY=$opensslPath/libcrypto.a \
+-DZLIB_FOUND=1 \
+-DZLIB_INCLUDE_DIR=$zlibPath \
+-DZLIB_LIBRARIES=$zlibPath/libz.a \
+-DSECP256K1_FOUND=1 \
+-DSECP256K1_INCLUDE_DIR=$secp256k1Path/include \
+-DSECP256K1_LIBRARY=$secp256k1Path/.libs/libsecp256k1.a \
+-DSODIUM_FOUND=1 \
+-DSODIUM_INCLUDE_DIR=$sodiumPath/src/libsodium/include \
+-DSODIUM_LIBRARY_RELEASE=$sodiumPath/src/libsodium/.libs/libsodium.a \
+-DMHD_FOUND=1 \
+-DMHD_INCLUDE_DIR=$libmicrohttpdPath/src/include \
+-DMHD_LIBRARY=$libmicrohttpdPath/src/microhttpd/.libs/libmicrohttpd.a
+
+
+test $? -eq 0 || { echo "Can't configure ton"; exit 1; }
+
+if [ "$with_tests" = true ]; then
+  ninja storage-daemon storage-daemon-cli blockchain-explorer   \
+  tonlib tonlibjson tonlib-cli validator-engine func fift \
+  lite-client pow-miner validator-engine-console generate-random-id json2tlo dht-server \
+  http-proxy rldp-http-proxy adnl-proxy create-state create-hardfork tlbc emulator \
+  test-ed25519 test-ed25519-crypto test-bigint test-vm test-fift test-cells test-smartcont \
+  test-net test-tdactor test-tdutils test-tonlib-offline test-adnl test-dht test-rldp \
+  test-rldp2 test-catchain test-fec test-tddb test-db test-validator-session-state
+  test $? -eq 0 || { echo "Can't compile ton"; exit 1; }
+else
+  ninja storage-daemon storage-daemon-cli blockchain-explorer   \
+  tonlib tonlibjson tonlib-cli validator-engine func fift \
+  lite-client pow-miner validator-engine-console generate-random-id json2tlo dht-server \
+  http-proxy rldp-http-proxy adnl-proxy create-state create-hardfork tlbc emulator
+  test $? -eq 0 || { echo "Can't compile ton"; exit 1; }
+fi
+
+strip storage/storage-daemon/storage-daemon
+strip storage/storage-daemon/storage-daemon-cli
+strip blockchain-explorer/blockchain-explorer
+strip crypto/fift
+strip crypto/func
+strip crypto/create-state
+strip crypto/tlbc
+strip validator-engine-console/validator-engine-console
+strip tonlib/tonlib-cli
+strip http/http-proxy
+strip rldp-http-proxy/rldp-http-proxy
+strip dht-server/dht-server
+strip lite-client/lite-client
+strip validator-engine/validator-engine
+strip utils/generate-random-id
+strip utils/json2tlo
+strip adnl/adnl-proxy
+
+cd ..
+
+if [ "$with_artifacts" = true ]; then
+  echo Creating artifacts...
+  rm -rf artifacts
+  mkdir artifacts
+  cp crypto/fift/lib artifacts/
+  cp -R crypto/smartcont/ artifacts/
+  cp build/storage/storage-daemon/storage-daemon artifacts/
+  cp build/storage/storage-daemon/storage-daemon-cli artifacts/
+  cp build/blockchain-explorer/blockchain-explorer artifacts/
+  cp build/crypto/fift artifacts/
+  cp build/crypto/func artifacts/
+  cp build/crypto/create-state artifacts/
+  cp build/crypto/tlbc artifacts/
+  cp build/validator-engine-console/validator-engine-console artifacts/
+  cp build/tonlib/tonlib-cli artifacts/
+  cp build/tonlib/libtonlibjson.0.5.dylib artifacts/libtonlibjson.dylib
+  cp build/http/http-proxy artifacts/
+  cp build/rldp-http-proxy/rldp-http-proxy artifacts/
+  cp build/dht-server/dht-server artifacts/
+  cp build/lite-client/lite-client artifacts/
+  cp build/validator-engine/validator-engine artifacts/
+  cp build/utils/generate-random-id artifacts/
+  cp build/utils/json2tlo artifacts/
+  cp build/adnl/adnl-proxy artifacts/
+  cp build/emulator/libemulator.dylib artifacts/
+  chmod +x artifacts/*
+  rsync -r crypto/smartcont artifacts/
+  rsync -r crypto/fift/lib artifacts/
+fi
+
+if [ "$with_tests" = true ]; then
+  cd build
+#  ctest --output-on-failure -E "test-catchain|test-actors"
+  ctest --output-on-failure
+fi