Feedback - Lab 1

Basic markdown formatting

While your report can be brief, please put some effort into formatting your text. Use headings, lists, code blocks, ... to make your report more readable. Check up on the GitLab markdown guide or use the built-in editor tools.

We want to emphasize formatting of code specifically, and the distinction between:

• Inline code: ./mvnw clean package
• Code blocks:

  gitlab-job:
    stage: build
    script:
      - echo "Hello, world!"
  

The same goes for all your issues, merge requests, wiki pages, etc. Any form of written media should have proper formatting. It is not only more pleasant to read, it also shows you put some effort into it.

Predefined variables

The ideal setup of your predefined variables would be:

variables:
  QUARKUS_CONTAINER_IMAGE_USERNAME: ${CI_REGISTRY_USER}
  QUARKUS_CONTAINER_IMAGE_PASSWORD: ${CI_REGISTRY_PASSWORD}
  QUARKUS_CONTAINER_IMAGE_REGISTRY: ${CI_REGISTRY}
  QUARKUS_CONTAINER_IMAGE_GROUP: ${CI_PROJECT_PATH}

Assuming our project is hosted at gitlab.stud.atlantis.ugent.be/myusername/devops-project, myusername being the project namespace, devops-project being the project name.

Some used CI_REGISTRY_IMAGE which resolves to gitlab.stud.atlantis.ugent.be:5050/myusername/devops-project as value for the registry. Jib will error here because myusername/devops-project is the group and it expects the top level registry here. The correct variable to use is CI_REGISTRY which resolves to gitlab.stud.atlantis.ugent.be:5050 in this case.

The best option for image group is CI_PROJECT_PATH as it combines both the project namespace and name, in our example myusername/devops-project. A more laborious but equally valid option would be to manually combine CI_PROJECT_NAMESPACE and CI_PROJECT_NAME with a / in between.

Some students used $GITLAB_USER_LOGIN/$CI_PROJECT_NAME instead, which in this case worked out because you are working in your personal namespace and you are the only one triggering pipelines. However, when we move this project to a group namespace, this will no longer work and when a collaborator triggers the pipeline, it will fail as well:

GITLAB_USER_LOGIN: The unique username of the user who started the pipeline, unless the job is a manual job. In manual jobs, the value is the username of the user who started the job.

Using predefined variables where possible

To avoid writing the same variables over and over again, it is best to use the predefined variables where possible. We can use the predefined variables from our container building job to link to our logic container in the run-game job. We could even define our own variables to make it more readable and easy to manage:

variables:
  ...
  # Using this variable we can easily change the registry for all our images if needed
  # This is not necessarily helpful in this case but we include it here as an illustration
  UTIL_REGISTRY: ${CI_REGISTRY}/utils/docker

run-game:
  stage: execute
  image:
    name: ${UTIL_REGISTRY}/devops-runner:latest
    entrypoint:
      - ""

  services:
  # Here we could also just use ${CI_REGISTRY_IMAGE}/logic-service:latest
    - name: ${CI_REGISTRY}/${CI_PROJECT_PATH}/logic-service:latest
      alias: logic-service

Store variables where they are needed

Variables can be defined globally or per job. Defining all the QUARKUS_CONTAINER_IMAGE_* variables globally is not necessary, as they are only needed in the maven-docker-generation job. Only move them to the global scope if they are used in multiple jobs and you are sure their presence won't affect other jobs in a negative way, an example:

If we would define QUARKUS_CONTAINER_IMAGE_BUILD: true and QUARKUS_CONTAINER_IMAGE_PUSH: true globally, it would be defined in all jobs. When we would later add a separate phase to our pipeline where we run mvn package (say we only want to build a JAR and deploy it somewhere or save it as an artifact), this command would build and push a container image as well, which is not what we want.

Always include MVN_CLI_OPTS in maven commands

In the build-game job we show you should use ./mvnw $MVN_CLI_OPTS compile, which contains the --batch instruction to make sure the build does not hang on user input and use defaults instead, it will also omit printing progress bars so you pipeline output isn't polluted.

Most students didn't think to include this option in the maven-docker-generation jobs and just called ./mvnw package or ./mvnw install without it. This is not a problem in this case because the build is not interactive, but it does result in a less readable log output. It is good practice to always include it when using mvn in a CI/CD pipeline.

Removing unnecessary debug output

We advised you to use a printenv and echo command to investigate the predefined variables, but you should remove this command after you are done with it. It is not necessary to keep it in your pipeline and it will only pollute your output. Same goes for other statements that were only necessary when debugging your pipeline.

Try to keep your finished pipeline as clean as possible.

Caching and artifacts

Most of you were able to add caching of dependencies and artifacts, which is great. The reason why we prefer using cache for dependencies and artifacts for build files is because of the way they are stored and how they can be used:

Cache

The cache is backed by storage close the runners, in our case the cache is a MinIO Object Store (S3).

Files in the cache can be used in subsequent jobs, but also across different pipeline runs. Meaning our first pipeline run will download all needed dependencies and store them in the cache. The next pipeline run will use the cache to fetch the dependencies, which is faster than downloading them from the internet.

This is why the cache has a key option. Our project can have different caches with different keys. A good default for a cache key is $CI_COMMIT_REF_SLUG or $CI_COMMIT_REF_NAME since these resolve into the branch or tag name for the current pipeline run. This way we can have a different cache for each branch or tag.

More examples of cache keys and common use cases can be found in the GitLab docs.

If we don't define a key, then our cache is global. This can lead to issues when you start working on multiple branches at the same time and these branches have differing dependencies. If for example someone is updating the dependencies in the main branch, but you are working on a feature branch, then your cache will be updated with the new dependencies as well. Your pipelines will still work, but they will have to download their dependencies again, which is not ideal.

Using a global cache

Many defined their cache globally, for their entire pipeline. However, the run-game job has no use for the cache, so downloading and checking it for updates will only slow down the pipeline.

Either define the cache explicitly for the jobs that make use of them, or define it globally and override the option in the jobs that don't need it like so cache: [].

Artifacts

Artifacts are only available in the current pipeline run. They are stored in the GitLab database and can be downloaded from the pipeline overview page. They are not available in subsequent pipeline runs (there are workarounds for this but they are not ideal).

Artifacts are best used to store build files, like JARs, WARs, etc. You can also use them to store logs, reports, etc. that you want to keep for a while.

Artifacts can be downloaded from GitLab, through Build > Artifacts or through the API. An expiration can be set so they only stay for a set period before they expire and are deleted. When moving onto releasing your software, these artifacts can easily be bundled into a release.

Artifacts and dependencies/needs

Jobs will download all artifacts from previous stages by default. Use of dependencies or needs enforces and controls which artifacts will be downloaded. Since our run-game job has no use for build files, we can prevent it from downloading the artifacts like so:

run-game:
  dependencies: []

Some used dependencies or needs to explicitly download the artifacts from the build-game job. This is not necessary, but it doesn't hurt to be specific. When your pipelines becomes more complex, it can be useful to explicitly define which artifacts are needed.

Not adding an alias for the service

In the run-game job, you need to add an alias for the service. This alias is used to reference the service in the LOGIC_URL variable. If you don't add an alias, the logic service will be only be reachable on its default hostname which is automatically constructed from the image URL (see GitLab docs on Accessing the services for more information)

Adding an alias overrides the default hostname and is as simple as adding the alias key to the service definition:

services:
  - name: ${CI_REGISTRY_IMAGE}/logic-service:latest
    alias: logic-service

Not adding the necessary dependencies

Some students had a pipeline run which was successful but didn't result in an image being pushed to their container registry. Looking in the job logs you can however spot these warnings:

[WARNING] [io.quarkus.config] Unrecognized configuration key "quarkus.jib.use-current-timestamp" was provided; it will be ignored; verify that the dependency extension for this configuration is set or that you did not make a typo
[WARNING] [io.quarkus.config] Unrecognized configuration key "quarkus.jib.base-jvm-image" was provided; it will be ignored; verify that the dependency extension for this configuration is set or that you did not make a typo

These warnings are caused by missing dependencies in the pom.xml file. The quarkus-container-image-jib extension is not included by default, so you have to add it yourself. This should have been set up in Building container images, where we build a container image locally. There we link to Quarkus docs which show the maven command to install the dependency. You can also add the dependency manually to the pom.xml file:

<dependency>
    <groupId>io.quarkus</groupId>
    <artifactId>quarkus-container-image-jib</artifactId>
</dependency>