Automated package building with the GitLab CI

development
packaging

(Luca Giambonini) #1

Dear all,
has been a while that I’m trying different tools to automate our building process. I tested antbs (the antergos build system) and was good, but very complicated to maintain, and some part of the code very specific for antergos OS.
So I decided to give a try to the gitlab CI (Continuous Integration). My interest started after reading this post: https://about.gitlab.com/2016/10/12/automated-debian-package-build-with-gitlab-ci/
why not do the same for chakra? in fact gitlab CI is an automated build server, conceived to build a single program, but in reality the system can call scripts one after the others.
Gitlab CI is based on a configuration file called .gitlab-ci.yml, were are defined all the build stages with its configurations. After lot of testing I decided to split the build process in 3 parts

  1. before_script
  2. build
  3. sign
  4. deploy

before_script (prepare.sh)

The whole process is based on the gitlab runner, this are machines that execute the Pipelines with its Jobs when triggered by the main gitlab process. We can consider them as slave, they do the job and report back the artefacts done. Different runners can build at the same time different packages, depending of the Tag assigned.
The runners are basically docker container that runs a chakra container (chakra-bootstrap)
The before_script is in charge to load our chakra continer and configure the basic environment. It consists to generate a builder user, initialize ssh and gpg agents and sync our repository.
The prepare.sh script will modify makepkg and the configuration in order to build with the chakra custom configurations (as already done with our build scripts).
Then the import-validpgpkeys.sh is called to parse the PKGBUILD and import the valid keys.

build (build.sh)

In order to build something you need a trigger. In our case I decided to build any PKGBUILD or .order file found in the commit files. I created some special scripts that can handle the build process. The gitlab ci interface is very clean, and the build process can be monitored directly on the webpage (or with the android app too :tada:)

How to trigger a build

is very simple! you have multiple choices:

  1. push a modified PKGBUILD
  2. push multiple modified PKGBUILDs
  3. push a .order file

How is build

the script build.sh is called by gitlab, this script will handle the whole build process, is divided in 3 main parts:

  1. list_packages
    the list of files that must be build is generated, every PKGBUILD found in the commit is added to the build array, every .order file is parsed and the list of pkgs added to the same build array
  2. sort_packages_by_dependency
    this function sort the build array in order to observe the dependency tree
  3. build_packages
    build the packages, what else?

sign

the operation above “build” generate some artefacts (*.tar.gz.xz). With the sign operation we sign those packages. The signature is the “Chakra Build Server”, and does not belong to any real user. I created a new private key and then imported in gitlab to be exposed when the pkg is signed.

deploy (deploy.sh)

as usual there is a script that list the packages and its signatures to be uploaded (rsync) on our server. Then through ssh the akbm script is called to sync the .db.
I decided to upload everything build automatically on [staging], then is the developer that has to move it in the proper repository.

##Current status:

  • :white_check_mark: building single or multiple PKGBUILD works
  • :x: building a .order list does not work, because the import-validpgpkeys.sh is not able to parse the PKGBUILD if there is an external reference (source, like for kf5/plasma group)
  • :white_check_mark: sign works with the custom Chakra Build server key (to be released with the dev team)
  • :white_check_mark: deploy on [staging] works

##Considerations:
For everyday use gitlab can be the solution, you upload a PKGBUILD and you have it done on [staging] (you can monitor the build process, and get notifications in case of failure). You can update packages with your smartphone, you have only to push the changes. And… can be your primary tool! (almost)
My concern starts with the “groups”, let me explain what I mean:
imaging that you want to update Plasma, you have to modify some PKGBUILD an then the .order/.sums/config files.
Consider now to push the .order file, and the build process will start automatically, but after some packages it fails. In order to fix the build you have to push a new commit (and include the .order file) and wait again that all the packages are build. This process can take lot of time, prepare-build-error-push-build-error-push-build… I know, is very rigorous, but is what we are looking for? any better idea how to handle that?
We can use the commit comment to trigger a group build! can be an alternative? otherwise we have to commit at every change the .order file, and to do that we need to modify something on it to be seen by git.
gitlab ci provides already a special key that can be used inside the git comment in order to skip the build, is [ci skip] or [skip ci]. We can use a similar approach…

Questions? you want to try it?
I enabled my testing repository to almost all chakra devs, if not, ask me the access.
The repository is: https://gitlab.com/AlmAck/chakra-build/

Please, any feedback is welcome, we are at the early stage, needs a lot of improvements!


(shainer) #2

Awesome!

As I mentioned on IRC, I am working on a Python script to connect Anitya to this continuous build system. Basically when you have a package monitored by Anitya, a message is sent every time a new upstream version is released; my script will listen for these messages and trigger an update of the package.

If we run Anitya on our server, this means anybody can have automated build & upload to staging of packages that do not usually require much manual intervention.

One question: how do we plan to test packages built through Gitlab? Ofc we can enable staging on our machine and upgrade frequently, but I am also thinking we can have some virtual machine or docker instance to do some standard test. Thoughts?


(Luca Giambonini) #3

Anitya can really semplify our lives if can automagically trigger a simple pkg update. I suppose this is applicable to a bugfix release and not for a new version. Is that true?

Yes, with gitlab we can add a “test” stage before the deploy to just install the pkg and verify it (there is a script to do this sanity check?)
But if we want to test the pkg globally then can be more complicated. I suggest to create a virtual machine/docker running chakra that every new pkg automatically test it. We can do that using a gitlab hook/trigger.

Note: the current gitlab CI implementation builds the packages against [testing]! we can change that but is important to know that each pkg must be moved manually from staging to testing. This give us the consciousness of what we are doing and what is done (or at least I hope so :sweat_smile:)

I would like to see other thoughts about this tipic gitlab and testing from our @team


(shainer) #4

By default Anitya triggers a new update when a new version is released upstream. I suppose you can craft a message that would trigger a simple rebuild (like when a dependency library is updated), but that would not be part of the service itself, you need to do it yourself.


(Luca Giambonini) #5

Ok, that’s fine. If upstream release a major release we have to touch the PKGBUILD file to be sure to include (or remove) the funtionalities (compile flags, new files …).
We need to put this infrastructure in place and let it run for a while, then we can understand the changes that we have to apply.


(Luca Giambonini) #6

“Tempus fugit”! after 6 months is time to continue this discussion.
Thanks to the migration of our repositories to gitlab we are able to test the CI environment directly with our packages.
As perhaps someone has already seen, the [gtk] repository was enabled to auto build packages.

I enabled only gtk in order to test the functionalities and let emerge any weakness or improvement on the system.

There is one major point that I want to discuss:

  • how do we define which repository the CI has to build against?

The current implementation build always against stable, and push the new packages on staging (this was the first approach, conservative)

My suggestion are:

  1. always build against testing, but the drawback is that we can’t prepare staging files or push an emergency fix on stable directly
  2. define a commit tag to force the repository, like “[stable] commit comment” to build against stable. (and upload to [stable] directly?)
  3. use the branches, master for stable, testing for testing… commit on the respective repository and then create a merge request.

other ideas? what do you prefer?


(Jeff Huang) #7

I prefer second option, don’t want to use git branch on our repository now, it’s a little messy.


(Neofytos Kolokotronis) #8

I prefer the 2nd also, so the packager can choose what is best for the situation. We could set one as default, but having the ability to define it in the commit is important.

There was an idea to drop branches all together, since we do not make much use of them, which makes sense.


(Luca Giambonini) #10

ok, let’s define in detail option 2)

  1. the default option is to build against testing, and when there is a security fix that must go do stable immediately use in the commit message “[stable]”.
    That can be applied to any other repo, staging, kde-staging and kde-unstable.
    With this solution we avoid to specify for each commit the repository from where it should be compiled.

  2. Use the default repository as defined today:
    core -> testing
    desktop -> stable
    gtk -> stable

we really build desktop apps against stable only? or we usually enable testing for taht?

I will go with option 1. at first, we can always change and improve that in the future.

Implemented:

It works like this:

If nothing is specified on a new commit then this rules are applies:
core and lib32 build against testing uploaded on testing
desktop and gtk build against stable uploaded on testing

To change that behaviour is possible to add in the git comment the repository which you want the package to be build, include the name inside the brackets [], for instance:
[staging] commit message

the name written in the git comment will be also the destination repository.


(Luca Giambonini) #11

Almost a month has passed, and so far the feedback was good. I would like to extend it even further, and enable an other repository. To me I would like to active desktop, the most active one, and make an other month of testing. After that we can extend to all repositories. Is ok for you? @team

I was asked to be able to push new commits with the tag [stable], so that the pkg is build and deployed directly in stable. IMHO, that’s a bit risky, because even if is a minor release the pkg can be corrupted or does not work.
I have this 2 proposals:

  1. we don’t allow deploying directly on stable (like now, build on stable, deploy on testing)
  2. add an other manual stage, that will move the pkg to stable

What do you think?


(Jeff Huang) #12

I prefer option 2, we still have some packages that can directly upload to stable :smiley:


(Luca Giambonini) #13

Ok, option 2) added but should be properly tested.

I introduced also the signature of the database after the deploy and the pkg move. That was a missing feature.


(Luca Giambonini) #14

No feedback so far?
I would like to enable [desktop] with gitlab-ci, any additional comments to improve the current situation?


(Jeff Huang) #15

It should be good, let’s go ahead :smiley:


(Francesco Marinucci) #16

I tested it with one package only, so I cannot give so much feedback :grimacing:


(Neofytos Kolokotronis) #17

I don’t have enough time currently to test and give feedback, but I would be happy to see this happening if no-one disagrees.


(Luca Giambonini) #18

Enabled gitlab-ci on desktop today, the procedure on how to use it is the same as gtk.

You can find the guide here:


(Luca Giambonini) #19

More then 2 months are passed since the last comment, I’m interested to get your feedback with your comments about the current implementation.

I have some ideas to improve the current status:

  • https://code.chakralinux.org/tools/chakra-gitlab-ci-scripts/issues/3 add a new stage that check the PKGBUILD before building and an other stage after the package is created. The pre-step can identify issues more quickly without building the package.
  • automate somehow the [testing]/[staging] tags in the comment, by adding a post git script (during push) to add that automatically. [skip-ci] will have the first priority over all others tags.
  • remove the source reference in our kf5/plasma/apps files to be able to build this groups automatically and in batch mode. A first try was prepared by Ram-Z and can be found here: https://code.chakralinux.org/packages/core/commit/5089e23c217dd753a4083a7ae5ca2cb711bb45ec In order to acomplish that we need a new script that is able to read the *.config file and update “sed” automatically the files listed in *.order list

What do you think about this new proposal above?
Do you have something to propose?


(system) #20

This topic was automatically closed 170 days after the last reply. New replies are no longer allowed.