The job is defined in integration/config.git in . It got written by/with @awight and @Ejegg . The bulk of it is to setup the test database, beside that it is all about cloning a few repositories using zuul-cloner and then running a customized phpunit command.
I am in holidays for the rest of the week. But I have added this task on my agenda for next week. Seems it is not going to be too much work to complete it.

I should probably do the first pass then present the result to @Ejegg and others from the fundraising team so they get familiar with the civicrm job? That will make it easier for the next changes needed.

Thanks @hashar, that sounds great! @Eileenmcnaughton is the local subject matter expert on CiviCRM and its build tools. Enjoy your holidays!

Change 516480 had a related patch set uploaded (by Hashar; owner: Hashar):
[wikimedia/fundraising/crm@master] wmf civicrm requires php-intl

https://gerrit.wikimedia.org/r/516480

Change 514011 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] civicrm job to Docker

https://gerrit.wikimedia.org/r/514011

Good news, I have managed to get the PHPUnit tests to pass in a local Docker container!

I have a question about the Civicrm buildkit. The current CI jobs fetch it from https://github.com/civicrm/civicrm-buildkit.git and I also noticed we have a Gerrit fork at wikimedia/fundraising/crm/civicrm-buildkit.git. Should CI uses the upstream one or should I switch to our Gerrit repository?

Note to self: if we use the Gerrit one, I would change the path to which the buildkit is cloned and update + the Docker container PATH statement.

I had a minor adjustement to do in :

- export AMPHOME="${WORKSPACE}/.amp-${BUILD_NUMBER}"
+ export AMPHOME="${XDG_CONFIG_HOME:-$WORKSPACE}/.amp-${BUILD_NUMBER}"

defaults to the user home directory but our CI Docker container runs as user which does not have a home dir (that is intentional).

is provided by Jenkins and it would no more be needed in the Docker container.

The , and environment variables were used to ensure we had a fresh start. With the Docker container they are no mo needed.

More self notes.

Rebuild the container:

git-review -d 514011
cd dockerfiles
docker rmi docker-registry.wikimedia.org/releng/civicrm:0.1.0; docker-pkg --info --conf docker-pkg.yaml build --select '*civicrm:*' --use-cache .

First tests run:

cd dockerfiles/civicrm
./example-run.sh

Next runs:

docker run --rm -it --volume /"$(pwd)"/cache://cache --volume /"$(pwd)"/log://log --volume /"$(pwd)"/src://src --env BUILD_NUMBER=fake_build_number --env BUILD_TAG=fake_build_tag --env JOB_ID=fake_job_id --env WORKSPACE='' docker-registry.wikimedia.org/releng/civicrm:latest 2>&1 | ts -s

Change 516484 had a related patch set uploaded (by Hashar; owner: Hashar):
[wikimedia/fundraising/crm@master] build: vary AMPHOME for Docker containers

https://gerrit.wikimedia.org/r/516484

Change 516485 had a related patch set uploaded (by Hashar; owner: Hashar):
[wikimedia/fundraising/crm@master] build: vary civicrm-build kit URL

https://gerrit.wikimedia.org/r/516485

Change 516487 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Experimental Docjer job for civicrm

https://gerrit.wikimedia.org/r/516487

Change 514011 merged by jenkins-bot:
[integration/config@master] docker: container for civicrm testing

https://gerrit.wikimedia.org/r/514011

Mentioned in SAL (#wikimedia-releng) [2019-06-11T13:17:16Z] <hashar> Building container docker-registry.wikimedia.org/releng/civicrm:0.1.0 # T210287

Change 516487 merged by jenkins-bot:
[integration/config@master] Experimental Docker job for civicrm

https://gerrit.wikimedia.org/r/516487

The new Docker based job can be tested by comment in Gerrit. To do the actual switch, I would need a couple patches for the civicrm CI entry point:

• https://gerrit.wikimedia.org/r/#/c/wikimedia/fundraising/crm/+/516484/ - adjusts AMPHOME
• https://gerrit.wikimedia.org/r/#/c/wikimedia/fundraising/crm/+/516485/ - use a different path for civicrm buildkit

The new Docker job passes on that change \o/

The version of the job is using the civicrm buildkit from Gerrit repository. Note the one from upstream github.

Change 516480 merged by jenkins-bot:
[wikimedia/fundraising/crm@master] wmf civicrm requires php-intl

https://gerrit.wikimedia.org/r/516480

Change 516484 merged by jenkins-bot:
[wikimedia/fundraising/crm@master] build: vary AMPHOME for Docker containers

https://gerrit.wikimedia.org/r/516484

Change 516485 merged by jenkins-bot:
[wikimedia/fundraising/crm@master] build: vary civicrm-build kit URL

https://gerrit.wikimedia.org/r/516485

Woohoo, nice work!

I can verify that as a person with no knowledge of the fr-tech stack ;-) I was able to build and run this image locally. Our PHPUnit tests ran and even passed, as advertised on the tin. The whole deal takes 6m45s (locally) which seems great for CI purposes.

One error log caught my eye, but seems to be harmless. I can't imagine the privileges on the /tmp db data directory matters a whit.

2019-06-12 23:35:02 140664573070720 [Warning] Ignoring user change to 'nobody' because the user was set to 'mysql' earlier on the command line
....
/usr/sbin/mysqld: One can only use the --user switch if running as root

We might eventually move these steps into the Dockerfile and the image rather than doing at runtime from . This change would require migration logic at the moment, so no rush to cash-in on the c. 10 seconds this might shave.

mkdir -p /tmp/mysqld/datadir
/usr/bin/mysql_install_db --user=nobody --datadir=/tmp/mysqld/datadir

Also in the future, this seems like a nice image to base an fr-tech development environment off of. We could create a docker-compose cluster that includes service dependencies like Redis, optionally a donationswiki for smoke-testing the full pipeline, etc. and exposes the CiviCRM web port. Prior art for inspiration: https://github.com/michaelmcandrew/civicrm-buildkit-docker

Change 516766 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Migrate wikimedia-fundraising-civicrm to Docker job

https://gerrit.wikimedia.org/r/516766

Change 516766 merged by jenkins-bot:
[integration/config@master] Migrate wikimedia-fundraising-civicrm to Docker job

https://gerrit.wikimedia.org/r/516766

In T210287#5255555, @awight wrote:

<snip>

One error log caught my eye, but seems to be harmless. I can't imagine the privileges on the /tmp db data directory matters a whit.

2019-06-12 23:35:02 140664573070720 [Warning] Ignoring user change to 'nobody' because the user was set to 'mysql' earlier on the command line
....
/usr/sbin/mysqld: One can only use the --user switch if running as root

The message comes from which in turns shells out to . The message is wrong, the command is actually running as and does not attempt to change ownership of the datadir which are already correct.

I think it I just a bug in the MariaDB version we use.

We might eventually move these steps into the Dockerfile and the image rather than doing at runtime from . This change would require migration logic at the moment, so no rush to cash-in on the c. 10 seconds this might shave.

mkdir -p /tmp/mysqld/datadir
/usr/bin/mysql_install_db --user=nobody --datadir=/tmp/mysqld/datadir

Definitely! I did pondered between having the container to create and spawn the database versus doing that in the crm/bin/ci* script. I have picked the container because it was easier for me and meant less changes in the repo.

The summary is does roughly:

• use run-with-mysql.sh entry point which initialize and starts a MySQL server
• run and
• vendor/bin/phpunit

We should be able to simply the entry point and move the logic to the crm repo with its testsuite being responsible for spawning a fresh MySQL database if none has been given to it. But eventually I went lazy and just copy pasted the logic from the previous job :]

Also in the future, this seems like a nice image to base an fr-tech development environment off of. We could create a docker-compose cluster that includes service dependencies like Redis, optionally a donationswiki for smoke-testing the full pipeline, etc. and exposes the CiviCRM web port. Prior art for inspiration: https://github.com/michaelmcandrew/civicrm-buildkit-docker

+1 :] Though on CI it is unknown how we can achieve that yet. The future would be to migrate to use a Helm chart and deploy the containers to a Kubernetes cluster.

For the purpose of Migrate wikimedia-fundraising-civicrm to a Docker container, I claim this task as resolved :]