Merge digitallumberjack/master modifications by digitalLumberjack · Pull Request #33 · sameersbn/docker-gitlab-ci-multi-runner

digitalLumberjack · 2017-01-29T09:35:53Z

Changes :

Runner upgraded to 1.10.4
Docker support (socket mode and dind mode)
Docker private registry support :
- Allows to use private images for jobs
Added RUNNER_DOCKER_ADDITIONAL_VOLUME support to share volumes between host and build containers
Added RUNNER_OUTPUT_LIMIT support to override the logs output limit.

You can use the docker executor by using `RUNNER_EXECUTOR=docker`. You must provide a docker image to use in `RUNNER_DOCKER_IMAGE` (e.g. docker:latest) If `RUNNER_DOCKER_MODE` is set to `socket`, the docker socket is shared between the runner and the build container. If it is not, you must use docker in docker service in your .gitlabci.yml definitions. See https://docs.gitlab.com/ce/ci/docker/using_docker_build.html for more info.

Upgrade to Ubuntu 16.04 LTS

…-multi-runner

sameersbn · 2017-01-29T16:03:08Z

Can you resolve the conflicts please?

digitalLumberjack · 2017-01-29T16:10:17Z

Should be ok. Sorry for the bad commit message i tried the online editor and i could not choose the message. I can amend if you need.

sameersbn · 2017-01-29T16:24:37Z

@@ -1,17 +1,33 @@
-FROM sameersbn/ubuntu:14.04.20170110
+FROM ubuntu:xenial


can you revert this?

sameersbn · 2017-01-29T16:25:33Z

+ && apt-get update \
+ && DEBIAN_FRONTEND=noninteractive apt-get install -y \
+      vim.tiny wget sudo net-tools ca-certificates unzip git openssh-client curl libapparmor1
+


by reverting to sameersbn/ubuntu:14.04.20170110 these lines will not be required

sameersbn · 2017-01-29T16:28:20Z

 COPY entrypoint.sh /sbin/entrypoint.sh
 RUN chmod 755 /sbin/entrypoint.sh

+RUN rm -rf /var/lib/apt/lists/*


is this line needed? I think it can be removed

sameersbn · 2017-01-29T16:31:59Z

@digitalLumberjack Lets resolve the issues in the PR before we hit the merge button.
p.s. i've added you as a collaborator to the project.

sameersbn · 2017-01-29T16:32:44Z

also i will give it a test before we merge this one. thanks for the contrib

inongogo · 2017-01-31T20:13:20Z

This is fantastic! I am eagerly looking forward to testing this! Thanks for all your work!

paolomainardi · 2017-02-01T14:17:00Z

Please merge it, this is a fantastic work!

digitalLumberjack · 2017-02-05T14:06:04Z

@sameersbn This should be ok now.
I bumped to 1.10.4 by the way.

davidwindell · 2017-02-23T10:47:22Z

LGTM? 👍

fwoelffel · 2017-03-09T16:17:25Z

Can this be merged?

Maescool

I'm running this version in production, upgraded from the current master without any hiccup, also tried the new features, works as advertised.

digitalLumberjack · 2017-04-19T08:44:39Z

I bumped to runner v1.11.2, @sameersbn as soon as you give your go we can merge this. Tell me if any modification is needed.

cpoetter · 2017-04-24T18:36:44Z

@digitalLumberjack @sameersbn Really awesome work you two! My Synology can't wait for this update :) Please press the merge button :D

BirgerK · 2017-06-02T09:01:27Z

Merging this update would be awesome! :)

davidwindell · 2017-06-28T11:21:18Z

@digitalLumberjack could the entrypoint detect a change to the RUNNER_DOCKER_IMAGE and update the config?

davidwindell · 2017-06-29T10:33:22Z

For those waiting for this to be merged, we've just switched all our builds over to digitallumberjack/docker-gitlab-ci-multi-runner:v9.3.0 and it's working grand. Great to finally be on the docker executor! :) Thanks @digitalLumberjack 👍

davidwindell · 2017-06-30T10:46:45Z

        gitlab-ci-multi-runner register --config ${GITLAB_CI_MULTI_RUNNER_DATA_DIR}/config.toml
    fi
+    if [[ -n ${RUNNER_CONCURRENT} ]];then
+      sed -i "s/concurrent = .*/concurent = ${RUNNER_CONCURRENT}/" ${GITLAB_CI_MULTI_RUNNER_DATA_DIR}/config.toml


@digitalLumberjack spelling mistake here, should be concurrent

Thanks I fixed that.

digitallumberjack/docker-gitlab-ci-multi-runner:v9.3.0-1 include this fix.

Maescool · 2017-07-27T10:29:55Z

I see @sameersbn has made @digitalLumberjack Collaborator, so you could merge this?

davidwindell · 2017-09-04T15:50:59Z

@digitalLumberjack could you bump to v9.5?

QuickJack · 2017-09-04T21:51:42Z

According to https://about.gitlab.com/2017/08/22/gitlab-9-5-released/ the old runner will stop working at September 22nd, 2017.

Please note, that this also affects the usability of docker-gitlab.

GarbageYard · 2017-09-26T04:31:22Z

@digitalLumberjack: I am getting following error while using the image you've shared:

./configure_proxy_artifactory.sh: line 13: docker: command not found

GitLab CI runner script:

docker run --name gitlab-ci-multi-runner -d --restart=always \
  --volume /var/run/docker.sock:/var/run/docker.sock \
  --volume /mnt/data/gitlab/gitlab-runner:/home/gitlab_ci_multi_runner/data \
  --env='CI_SERVER_URL=https://gitlab.dev.abc.net/ci' --env='RUNNER_TOKEN=RijUZPnMjGeNF2JYt' \
  --env='RUNNER_DESCRIPTION=runnerA' --env='RUNNER_EXECUTOR=docker' \
  --env='RUNNER_DOCKER_IMAGE=docker:17.07.0-ce' --env='RUNNER_DOCKER_MODE=socket' \
  --link gitlab.dev.abc.net \
  digitallumberjack/docker-gitlab-ci-multi-runner:v9.3.0-1

.gitlab-ci.yml content:

image: node:6.11

types:
        - build
        - deploy

build_app:
        type: build
        script: "bash chmod +x ./configure_proxy_artifactory.sh"
        script: "bash ./configure_proxy_artifactory.sh"

From within the container, here's my observation:

root@5489eb3ebe42:/home/gitlab_ci_multi_runner# ls -l /var/run/docker.sock
srw-rw---- 1 root docker 0 Jul 21 00:53 /var/run/docker.sock
root@5489eb3ebe42:/home/gitlab_ci_multi_runner# docker ps
bash: docker: command not found

Any idea how i can fix this?

Maescool · 2017-09-26T22:51:51Z

@GarbageYard You are using the wrong image for this kind of setup
you need at least the following in your .gitlab-ci.yml

image: docker:latest
services:
  - docker:dind

in the Dockerfile you use to build the image, you should refer to the node image.

GarbageYard · 2017-09-27T04:25:43Z

Thanks for the reply @Maescool! Sorry, but i am confused now because if i'm going to use image: docker:latest, then how will i get the node app? Will i be using some package manager inside the .gitlab-ci.yml file to download the node installer?

Maescool · 2017-09-27T19:43:50Z

@GarbageYard so, there are a couple of parts.
if you want to build a docker image using gitlab-ci-multi-runner,
in my case I use docker in docker, so that's why I have the services thingie, you might not need it.
you have your .gitlab-ci.yml something like this

image: docker:latest

services:
  - docker:dind

variables:
  GIT_SUBMODULE_STRATEGY: recursive
  CONTAINER_TEST_IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME
  CONTAINER_RELEASE_IMAGE: $CI_REGISTRY_IMAGE:latest

before_script:
  - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY

stages:
  - build
  - release

build:
  type: build
  script:
    - docker build --pull -t $CONTAINER_TEST_IMAGE .
    - docker push $CONTAINER_TEST_IMAGE

release-image:
  stage: release
  script:
    - docker pull $CONTAINER_TEST_IMAGE
    - docker tag $CONTAINER_TEST_IMAGE $CONTAINER_RELEASE_IMAGE
    - docker push $CONTAINER_RELEASE_IMAGE
  only:
    - master

(this is basically by default gitlab-ci build file to build docker images)

Then you have your Dockerfile, that looks something like this

FROM node:6.11
COPY . /usr/src/app

The entrypoint stuff etc.. idk with node, and for your project, this is just an example of how I do my image builds.

GarbageYard · 2017-10-02T14:26:13Z

Hi @Maescool! I tried using DIND but somehow it's failing to retrieve the image via Artifactory (our private container registry).

Runner log error:
Running with gitlab-ci-multi-runner 9.5.0 (413da38)
  on RunnerA (d8ed43a6)
Using Docker executor with image docker.artifactory.abc.net/docker:17.07 ...
Starting service docker.artifactory.abc.net/docker:17.07-dind ...
Pulling docker image docker.artifactory.abc.net/docker:17.07-dind ...
ERROR: Preparation failed: Error response from daemon: Get https://docker.artifactory.abc.net/v2/: x509: certificate signed by unknown authority

I guess it's because it's unable to login to Artifactory first. I saw this link but it seems it's for images and not for services. I still tried configuring Secret Variable (DOCKER_AUTH_CONFIG) but it's not working.

Content of .gitlab-ci.yml

image: docker.artifactory.abc.net/docker:17.07

variables:
  DOCKER_HOST: tcp://docker:2375

# This before_script block was added later but it seems this block
# isn't executed before the DIND tries fetching image from Artifactory
before_script:
  - docker login -u svc-art-user -p some-pwd docker.artifactory.abc.net
  - docker info

services:
- docker.artifactory.abc.net/docker:17.07-dind

build:
  stage: build
  script:
  - docker build -t my-docker-node-image .

All (GitLab, Runner & DIND) containers are on one host. I am able to docker login docker.artifactory.abc.net -u svc-art-user -p some-pwd successfully. Even without DOCKER_AUTH_CONFIG, my image (shown above i.e., docker.artifactory.abc.net/docker:17.07) was getting fetched without any issue. It seems it's because it was using my host's ~/.docker/config.json. If that's the case, why isn't the service (docker.artifactory.abc.net/docker:17.07-dind) also using the same credentials?

digitalLumberjack and others added 16 commits October 22, 2016 11:16

Upgrade gitlab-ci-multi-runner to v1.7.0

1fede7b

Merge remote-tracking branch 'origin/release/v1.7.0'

d3df8a7

added docker private registry support

21bdf60

fixed auth for docker

c78eeff

update to xenial

ee2edcf

git-core -> git

0de46ba

Merge pull request #1 from SaschaNaz/xenial

dc8b3bf

Upgrade to Ubuntu 16.04 LTS

gitlab-ci-multi-runner: upgrade to 1.8.0

89a554b

Merge branch 'master' of https://github.com/Maescool/docker-gitlab-ci…

225b987

…-multi-runner

bumped to v1.9.0 of gitlab runner

97d5e51

added support of RUNNER_CONCURRENT to set parallels jobs

30865ea

added additonal volumes

9369254

fix reboot error where mkdir already existing directory

688b056

bumped to 1.9.3 and added support of RUNNER_OUTPUT_LIMIT

974e3f2

bumped to runner v1.10.2

a7cace0

digitalLumberjack mentioned this pull request Jan 29, 2017

Updates? #22

Open

sameersbn requested changes Jan 29, 2017

View reviewed changes

digitalLumberjack added 4 commits February 5, 2017 11:28

rollback to sameersbn/ubuntu:14.04.20170110 base image

9947cf3

bumped to gitlab runner v1.10.4

3570012

Merge remote-tracking branch 'sam/master' into mergemaster

711f75d

restored origin image name

fd83ccf

digitalLumberjack force-pushed the master branch from 7e20562 to 3570012 Compare February 5, 2017 11:00

fixed package installation

e3a7ab6

added RUNNER_AUTOUNREGISTER variable to unregister the runner on stop

71fbf3d

bumped to 1.11.0

6c55ae8

Maescool approved these changes Apr 18, 2017

View reviewed changes

Maescool mentioned this pull request Apr 18, 2017

gitlab-ci-multi-runner: upgrade to 1.8.0 #31

Closed

bumped to runner v1.11.2

80abfc5

cpoetter mentioned this pull request Apr 28, 2017

Add tutorial for Synology setup sameersbn/docker-gitlab#1196

Closed

digitalLumberjack mentioned this pull request May 18, 2017

Add Kubernetes Executor support digitalLumberjack/docker-gitlab-ci-multi-runner#2

Open

davidwindell suggested changes Jun 30, 2017

View reviewed changes

fixed spelling on concurrent parameter

d050d92

QuickJack mentioned this pull request Jul 24, 2017

API Deprecation sameersbn/docker-gitlab#1299

Closed

QuickJack mentioned this pull request Oct 11, 2017

Runner not working anymore sameersbn/docker-gitlab#1377

Closed

		@@ -1,17 +1,33 @@
		FROM sameersbn/ubuntu:14.04.20170110
		FROM ubuntu:xenial

Conversation

digitalLumberjack commented Jan 29, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sameersbn commented Jan 29, 2017

Uh oh!

digitalLumberjack commented Jan 29, 2017

Uh oh!

sameersbn Jan 29, 2017

Choose a reason for hiding this comment

Uh oh!

sameersbn Jan 29, 2017

Choose a reason for hiding this comment

Uh oh!

sameersbn Jan 29, 2017

Choose a reason for hiding this comment

Uh oh!

sameersbn commented Jan 29, 2017

Uh oh!

sameersbn commented Jan 29, 2017

Uh oh!

inongogo commented Jan 31, 2017

Uh oh!

paolomainardi commented Feb 1, 2017

Uh oh!

digitalLumberjack commented Feb 5, 2017

Uh oh!

davidwindell commented Feb 23, 2017

Uh oh!

fwoelffel commented Mar 9, 2017

Uh oh!

Maescool left a comment

Choose a reason for hiding this comment

Uh oh!

digitalLumberjack commented Apr 19, 2017

Uh oh!

cpoetter commented Apr 24, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

BirgerK commented Jun 2, 2017

Uh oh!

davidwindell commented Jun 28, 2017

Uh oh!

davidwindell commented Jun 29, 2017

Uh oh!

davidwindell Jun 30, 2017

Choose a reason for hiding this comment

Uh oh!

digitalLumberjack Jul 3, 2017

Choose a reason for hiding this comment

Uh oh!

Maescool commented Jul 27, 2017

Uh oh!

davidwindell commented Sep 4, 2017

Uh oh!

QuickJack commented Sep 4, 2017

Uh oh!

GarbageYard commented Sep 26, 2017

Uh oh!

Maescool commented Sep 26, 2017

Uh oh!

GarbageYard commented Sep 27, 2017

Uh oh!

Maescool commented Sep 27, 2017

Uh oh!

GarbageYard commented Oct 2, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

13 participants

digitalLumberjack commented Jan 29, 2017 •

edited

Loading

cpoetter commented Apr 24, 2017 •

edited

Loading