This is so that we can do simple changes to the docker images
in provisioning, without needing to update the tag here.
If a backwards-incompatible change in the docker images needs to be committed
in provisioning and here, it is possible because the images retain their old
unique SHA1 tag, in addition to being tagged as latest. See comment for more
details.
Requires the change in qt5 repository, with commit sha:
e4f9ac5607a329bae045567a339d36469bc4fff6
Task-number: QTBUG-79867
Change-Id: I1bc72edec62487530575d7e113a25afe16d09129
Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
This change is used to generalize a template docker-compose file for all
modules. Ideally, the leaf module only need to keep a docker compose
file for all platforms (docker-compose.yml).
NOTE:
The version of docker-compose file downgrades from 3.4 to 2.1 because
the 'extends' keyword is not supported in Compose version 3.x.
Change-Id: I2e36fd9236eda86cb5fcf940d787ccefe9200696
Reviewed-by: Maurice Kalinowski <maurice.kalinowski@qt.io>
Sometimes, it fails in tst_QFtp::proxy or tst_QFtp::activeMode under the
stress test. It complains about "Network operation timed out" on
vsftpd.test-net.qt.local.
When this issue happens in tst_QFtp::proxy, it shows "USER-ERR reject:
172.18.0.5 (ForkLimit 40)" in the log of ftp-proxy. By default, the
ftp-proxy only supports 40 incoming client connections per minute. To
make the ftp-proxy more powerful, this change extends the limits from 40
to 2000.
When this issue happens in tst_QFtp:activeMode, vsftpd shows 426 Failure
writing network stream error. It is a known issue of vsftpd. A quick fix
is turning use_sendfile off.
Change-Id: Iad50469654041bf30f92ef00805034f0d4aa9c3f
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Timur Pocheptsov <timur.pocheptsov@qt.io>
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
The proxy_data() defined in tst_qftp.cpp expects five items (bigfile,
nonASCII, rfc3252, rfc3252.txt, and upload) in the server folder
(ftp/qtest). The file rfc3252 and nonASCII folder were missing.
Change-Id: I995d6e254875ade22a1def53187077f1cc8d4c98
Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
Reviewed-by: Timur Pocheptsov <timur.pocheptsov@qt.io>
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
In order to reuse the test server to the external modules, it is much
easier to share the common configurations (scripts) and test data via
Dockerfile. In addition, the external module can create more layers
depending on their needs. Therefore, supporting multi-stage builds is
needed. The disadvantage is that the docker-compose needs to re-build
the images every time. However, it is just a one-time effort. If the
Dockerfile doesn't get changed, the extra build time can be ignored.
Because of multi-stage builds, the test server will keep a Dockerfile at
least. Therefore, the volume sharing is no more needed. The test data of
a service can be added into the images by using COPY/ADD commands.
NOTE:
This patch relies on docker-compose v1.21.0 (docker-compose build now
supports the use of Dockerfile from outside the build context).
Change-Id: Ib3f6a5fcf6979732ae8a40a494a1360fca4ac7bf
Reviewed-by: Maurice Kalinowski <maurice.kalinowski@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
The docker-compose files were trying to use volume sharing, which is not
supported on Windows in conjunction with docker-machine.
Hence create a separate layer on Windows, which copies the configuration
files to the target.
Change-Id: Ifeacc56198ffc8fb2eb31c14ab91334e22e916f5
Reviewed-by: Maurice Kalinowski <maurice.kalinowski@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Currently docker is only used for the "echo" test, and then only the
echo server. The SOCKS5 tests have been disabled from when our current
test server was new.
Change-Id: I21b0c5f10e722a42a0880b2500d1bcbc609a03bf
Reviewed-by: Timur Pocheptsov <timur.pocheptsov@qt.io>
The iptables container launches with extra capabilities to actually be
able to make changes to the tables.
Change-Id: I892fd18853ce882709e21791e6c88217e5029d53
Reviewed-by: Timur Pocheptsov <timur.pocheptsov@qt.io>
and make it work with our new docker-based test server
Change-Id: I98b5b5b1e2cdca46b7f15be72aa1483d9455403d
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
to make it work with our new docker-based test server.
Change-Id: I76345a2d3d768b8a571f2c85e69f6a21e9a96d7e
Reviewed-by: Ryan Chu <ryan.chu@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
On Windows and macOS, the containers are deployed into a virtual
machine using the host network. All the containers share the same
hostname (qt-test-server), and they are connected to the same network
domain (local).
When running test in such platforms, use the single-name SSL certificate
(qt-test-server.local) for SSL related tests.
Change-Id: Idf33e01e8dd8814510d848b87b59b5fc0edc903e
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@qt.io>
Setting the fall-back network interface of danted's environment to eth0
by docker-compose file is redundant because the value of danted's
configuration (danted.conf) has been set to eth0 by default.
Change-Id: If2dea8daaf851577a573e201e9c50684916e5206
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@qt.io>
For macOS and Windows, the test-server containers are deployed in the
VirtualBox (Boot2Docker). Because Boot2Docker is a lightweight Linux
distribution made specifically to run Docker containers, it doesn't
install avahi-daemon and support mDNS discovery.
To resolve this problem, Docker compose file supports "extra_hosts" to
add hostname mappings inside containers.
BFAIL items:
tst_QNetworkReply::headFromHttp(...+proxy...) (ten cases)
tst_QNetworkReply::ioGetWithManyProxies(http-on-http)
tst_QNetworkReply::ioGetWithManyProxies(http-on-http2)
tst_QNetworkReply::ioGetWithManyProxies(http-on-multiple-http)
tst_QNetworkReply::ioGetWithManyProxies(http-on-http+socks)
tst_QNetworkReply::ioGetWithManyProxies(http-on-ftp+http+socks)
tst_QNetworkReply::ioPostToHttpFromSocket(...+proxy) (twelve cases)
tst_QNetworkReply::ioPostToHttpFromSocket(...+proxyauth) (ten cases)
Change-Id: Iec55966a9b5f191b7446985a15b49a8b09dcf407
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@qt.io>
Docker compose file supports variable substitution. When running
docker-compose up, Compose looks for the environment variables from
shell and substitutes the values at runtime.
Change-Id: I5255ead82276fac7db24ee74af453f83ca20bbe6
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
There is no docker bridge on Windows. Docker document recommends using
port mapping to connect to a container. The problem is that it causes a
port conflict if the user is running a service that binds the same port
on the host. This change applies the same solution of macOS to deploy
the docker environment into VirtualBox and use the host network option.
Task-number: QTQAINFRA-2294
Change-Id: Iedcb8daa39373f02adb59f02eae2775f02870c54
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
This appears to be a timing issue of vsftpd's initial script
(/etc/init.d/vsftpd). When starting the vsftpd service, the script uses
a while loop to make sure that the vsftpd process has been created with
the expected PID. The problem is that it should sleep if it can't grep
the same PID via ps command.
while [ ${n} -le 5 ]
do
_PID="$(if [ -e ...vsftpd.pid ]; then cat ...vsftpd.pid; fi)"
if ! ps -C vsftpd | grep -qs "${_PID}"
then
break
fi
sleep 1
n=$(( $n + 1 ))
done
if ! ps -C vsftpd | grep -qs "${_PID}"
then
log_warning_msg "vsftpd failed - probably invalid config."
exit 1
fi
However, synchronization based on time is not working in general. This
change will rewrite the while loop to an infinite loop and remove the
exclamation symbol (!) from the if condition.
Upgrading the version of vsftpd is not helpful here because
vsftpd_3.0.3-11 (Ubuntu 18.10) didn't resolve it yet.
Change-Id: I07382709c33bd9bab61fcea76ab7deca5f630084
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
This update is used to free the dependencies of the specific Ubuntu
packages. It ensures that test server is using the latest version of the
Ubuntu packages to test network changes.
Change-Id: I3257f435e6da02e3c6d5a141ece9c5d025e13065
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@qt.io>
There is no docker bridge on macOS. Docker document recommends using
port mapping to connect to a container; but it causes a port conflict
if the user is running a service that binds the same port on the host.
An alternative solution is to deploy the docker environment into
VirtualBox and use the host network option.
Task-number: QTQAINFRA-2293
Change-Id: I05dc65c5f8b4be7a1b1874a4ec7c034cc68679ca
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@qt.io>
The internal and external keywords in the server configuration are used
to specify the network interface (IP address) that the Dante server
should bind to. Because docker-compose allows a container to join more
than one network (by adding multiple entries under the top level
networks key to a docker-compose file), the default server
configuration (danted.conf) shall be updated in accordance with the
settings in the docker-compose file.
The user can pass the setting of a docker-compose file to containers by
adding environment variables in the docker-compose file. After the
container gets created, the danted.sh script uses the input environment
variables to overwrite the default value of internal and external
interfaces before starting the Dante server.
environment:
- danted_internal=eth1
- danted_external=eth1
- danted_auth_internal=eth1
- danted_auth_external=eth1
Change-Id: I7c65cdd136ba5dac387bee5e22e1f4335e960101
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@qt.io>
This change frees the tests of their dependence on the Qt internal test
server (qt-test-server.qt-test-net). It makes the developers run the
tests out of Qt testing infrastructure.
If the user has installed Docker engine on their host, the test servers
will be built up inside separate Docker containers, and then, the test
case goes with the Docker-based test servers. Otherwise, the test case
will keep using the Qt internal test server.
Task-number: QTQAINFRA-1686
Change-Id: I518bc3675bfd658938509744b0e7e0610bc8bf66
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@qt.io>
To resolve the challenge of connecting Docker containers from the host
without extra user-efforts, Avahi is used to publish the Docker-based
test servers (Docker containers) on a local network. It is a free
Zeroconf implementation, and it provides multicast DNS (mDNS) service
discovery. By default, mDNS only resolves hostname ending with the
".local" domain. In this change, the domain name of test servers is
changed to "test-net.qt.local". As stated in the standard, the DNS query
for a local domain will be sent to the mDNS multicast address. It avoids
the need for root access to modify system files.
Task-number: QTQAINFRA-1686
Change-Id: I255fba0720ee6d0e84c9cb5ca9289d4466578922
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Ryan Chu <ryan.chu@qt.io>
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@qt.io>
The existing network test server has some limitations. Most notably, it
is not accessible by every Qt developer. Also, some services don't allow
simultaneous access, which causes flaky test results.
Instead of centralizing all the services to one physical machine, the
idea is to build up several dedicated servers inside separate Docker
containers.
1. Create testserver.pri and integrate it into the make check command of
Qt Test.
2. Define QT_TEST_SERVER flag for changing test parameters at compile
time.
Task-number: QTQAINFRA-1686
Change-Id: I0422ddb97eb8c11b4818771454851d19671253b1
Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@qt.io>
Reviewed-by: Ryan Chu <ryan.chu@qt.io>
