How to use the Chakra Build System (deprecated in favour of GitLab wiki)

This is an unlisted, ad hoc draft, to act as a placeholder in order to provide a link that can be used in other articles and a place to define what content is in scope or not.

If you’re a user looking for the actual information presented with proper formatting, please see the wiki.chakralinux.org article on the same topic. This draft will most likely remain difficult to read for some time due to the lack of structure and proper formatting.


The Chakra Build System is a chroot containing the software needed to create and upgrade Chakra packages, i.e. compile a package from C++ sources into binary executables (in many cases).

Here you can learn how it works and how to use it to maintain any official repository.


Contents


Features

  • Chrooted build environment using systemd-nspawn.
  • Clean and separated. ‘’‘No’’’ changes on the build machine are needed, just one new directory will be created.
  • Transparent creation of separate packages with debug symbols.
  • Runs on any distribution that has Filename|pacman.static installed.
  • Automatic dependency handling for depends and makedepends.
  • Does not break functionality. All standard makepkg stuff works, like repackaging for example.
  • Flexibility, like creating a complete KDE package set with just one command (from tarballs or SVN tag/trunk, even if there is no KDE installed yet).
  • Automatic repositories handling (creating/updating databases, cleaning/moving packages, atomic uploads).
  • Install/reinstall/uninstall complete chroots with just one command.
  • Simple. Not over-engineered but clear, straight and understandable scripts, even for newbies.
  • A nice setup to get started in almost no time.

Setup

Clone the repository

First you need to clone the repository. Make sure you are in a filesystem that has at least around 10GB of free space for the various chroots.

$ git clone git://git.chakralinux.org/buildsystem.git

Note: Make sure that you have git installed.

Create a chroot

Enter the buildsystem folder.

$ cd buildsystem

Now create a new chroot using make_chroot.sh and one of the provided configuration files in conf/.

$ ./make_chroot.sh conf/stable.conf

Note: You can see into conf/ to find what configuration files can be used, for now, we have stable.conf for [desktop] and [gtk]. testing.conf is used for [core]. For [lib32], please use lib32-testing.conf instead of lib32-stable.conf because we use testing branch instead of master branch. For [unstable], we are working on it, stay tuned!

Note: If you don’t have the rights to push to our repositories, use the -n option.

Note: Read ./make_chroot.sh -h for more information.

We provide a bunch of default config files but you can write your own. They are simply pacman.conf files, for more information read man pacman.conf.

The script will ask for your user password. Remember your user must have sudo-rights.

$ ./make_chroot.sh conf/stable.conf
 :: Running as user: jsmith
 :: Running on Chakra: 2015.03
 :: Using pacman
 :: Root                 : chroots
 :: Chroot Installation  : chroots/stable 
 :: Do you want to continue? [Y/n]

Check that the paths are correct and enter y and Enter to continue, and the scipt will install all the required packages.

If you didn’t run the script with the -n option, you will be then asked for your SSH password.

 :: Enter your git crendentials
 :: Full name (e.g. 'John Smith'):
John Smith
 :: E-mail (e.g. 'jsmith@chakralinux.org'):
jsmith@chakralinux.org
 :: Enter your ssh username:
jsmith

And that’s it! You have a chroot created and prepared to manage a repository. You can repeat the process for a different configuration.

Even if there were errors during the process, like no internet connection or wrong password, the script will continue until the end. That doesn’t mean everything is okay. If there were errors, you may need to remove the chroot and try again.

This is a series of steps that should be done to chroots you have just created.

Sync the local database:

$ sudo pacman -Syu

If you are building C packages for the lib32 repo, you need to install lib32-glibc and lib32-gcc-libs, or gcc will not work in 32-bits mode.

Note: should be done automatically by the script.

$ sudo pacman -S lib32-glibc lib32-gcc-libs

Understanding the shell prompt

Once inside the buildsystem, you will be prompted by a shell similar to this one:

tetris4@stable: ~/desktop (master *=)]$

*’’‘tetris4’’’ is your ‘’’’‘username’’’’’
*’’’@stable’’’ is the ‘’’’’.conf configuration you used’’’’’
*’’’~/desktop’’’ is your ‘’’’‘working directory’’’’’
*’’‘master’’’ is the ‘’’’‘git branch’’’’’

  • ‘’’*’’’ indicates ‘’’’‘untracked or modified files’’’’’ locally and is followed by one of these:
    ** ‘’’=’’’ indicates your branch is ‘’’’‘equal to upstream’’’’’
    ** ‘’’>’’’ indicates your branch is ‘’’’‘ahead of upstream’’’’’
    ** ‘’’<’’’ indicates your branch is ‘’’’‘behind upstream’’’’’
    ** ‘’’<>’’’ indicates your branch has ‘’’’‘diverged from upstream’’’’’ (is at the same time behind and ahead)

Understanding the directory layout

==File Hierarchy==
{|
| Filename|make_chroot.sh
|Script to create a chroot. It takes a configuration file from the Filename|conf/ folder as argument. Run ic|./make_chroot.sh -h for more info.
|-
| Filename|enter_chroot.sh
|Script to enter a chroot. It takes a repository name as argument, and optionally a branch. Run ic|./enter_chroot.sh -h for more info.
|-
| Filename|chakra/

This will be mounted as your home directory inside the chroot.
Filename
-
Filename
User configuration.
-
Filename
It contains all the scripts needed for upload, remove or move packages to the server.
-
Filename
Is where packages go once they are compiled using the build system.
-
Filename
When building a package, sources are downloaded here, so they can be shared by different architectures.
-
Filename
Contains pacman configuration for repositories and more.
-
Filename
Contains cached packages.
-
Filename
Directories for the different combinations.
-
Filename
Contains patches for our version of ic
}

After [[#Enter a Chroot|entering a chroot]], the Filename|chakra/ directory will be mounted as your home directory in Filename|/chakra.


Tasks

Enter a chroot

First, enter into the Filename|buildsystem/ folder:
$ cd builsystem

Then run this for desktop:
$ ./enter_chroot.sh desktop

Note| Read <code>./enter_chroot.sh -h</code> for some more information.

Or this for the testing git branch of desktop:
$ ./enter_chroot.sh desktop testing

You will be asked for your sudo password. (it might still be cached)

Upon entering the chroot, git will fetch the newest changes from our repos and display wether your local clone is ahead or behind upstream (or both for that matter). If either is true make sure to sync the repositories before doing any kind of work.

To ensure your chroot is clean and synced before starting to build and add packages, a good practice is to always do this first after entering your chroot:
$ git pull
$ pacman -Syu

Tip| You can view the current remote URL with:
$ git remote show origin 

and set a new one with
$ git remote set-url origin

Exit a chroot

Just run this from inside the chroot:
$ exit

Or hit Key|Ctrl+D.

Add a Chroot to your Build System

The steps to create a new chroot into a current build system is pretty much the same than the one to setup the build system with one initial chroot. Just choose a different config files.

Add a New Package

Create a Build Package

To create a new package, read the [https://wiki.archlinux.org/index.php/Creating_Packages Arch Wiki], and make sure your follow link|Packaging Standards|Chakra standards.

If you are porting a build package from the link|Chakra Community Repository, check link|Packaging Standards|standards too, since there are some differences between packages in the CCR and packages in the link|Repositories|official repositories.

Once you have the Filename|PKGBUILD and other needed files ready, put them in a folder with package’s name, and put that folder into the right Filename|buildroot/’’’’‘repository’’’’’-’’’’‘branch’’’’’-’’’’‘architecture’’’’’/pkgbuilds/ folder.

Build the Package

Now you can [[#Enter a Chroot|enter your chroot]]. Then get into the package folder and run link|Makepkg:
cd <package_name>
makepkg -sr

Clean the Package Directory

{| style=“float: right; text-align: center;” border=1px cellspacing=0
|’’’:cp’’’
’‘is alias for’’
’’‘rm pkg src dbg hdr -r’’’
|}
Once compiled, clean ‘’’’‘package_name’’’’’ folder, and just leave the Filename|PKGBUILD file and any Filename|.install file, Filename|ChangeLog or patches. That is, the files that was already in the folder before running link|Makepkg. Then check what additional files are there or what have you changed:
$ git status

There should be no additions, since you have already deleted those files created during the build process. If there is any unexpected untracked file, remove it and run git status again.

Note that when a package successfully compiles, it is moved to Filename|/chakra/_repo/, and the Filename|.pkg.tar.xz file in package’s folder is just a symlink that Git will ignore. That means, you don’t need to delete it (but can do it).

Test the Package

If you are working on a package for the same branch and architecture as yours, you can and should test it in your own system before submitting it to the Git or the Rsync repositories.

[[#Exit a Chroot|Exit your chroot]], go to Filename|chakra/_repo//, and install it with:
$ pacman -U path/to/packagefile.pkg.tar.xz

Then run it in your system, and once you have checked it works as expected, jump to the next step.

Package Chains

Package chains are lists of packages you might need to build at the same time and in a specific order. These are defined in configuration files (.conf, .order) that exist in the repository folder, together with all the package folders that contain the PKGBUILDs.

  • the ‘’’.conf’’’ files include parameters that are common for all the packages that refer to them and can be defined before running a build.
  • the ‘’’.order’’’ files include a simple list of the packages that need to be build in a group, ordered according to their dependencies (if package A depends on package B, B should be above A on the list).

Some frequently used .order files include plasma, kdeapps and kde-telepathy for the desktop repository and frameworks for the platform repository. For example:

  • For the Plasma group one should run inside a desktop chroot environment:
    $ build.sh plasma.order
  • For the Frameworks one should run inside a platform chroot environment:
    $ build.sh frameworks.order

You can create your own custom .order and .conf files if you wish and run them through build.sh.

==Maintenance==
===Update===
It is important to keep your buildsystem up to date with the latest changes, so it is recommended to regularly update it by entering the buildsystem folder and git pull.

$ cd buildsystem
$ git pull

==Official Packagers′ Zone==
The following guidance would introduce you on how to package officially.

The guidance is written for two main reasons:

  • Pacman is designed to use detached PGP signature for package verification, thus we must sign our package during packaging.
  • We use SSH key authentication on remote servers.
  • Out git server supports SSH access only

===Introduction===

As Link|How_to_become_a_packager|granted by our team, you would have write access to our Git repo and package repo, as well as community-supported remote server for packaging.

The steps include system preparation for signing and standard package manipulation procedure.

===System Preparation===
====SSH Agent====

For generation of ssh key pair, please refer to [https://wiki.archlinux.org/index.php/SSH_keys#Generating_an_SSH_key_pair| Archwiki’s introduction]

The buildsystem is designed to use your running ssh-agent and use the keys stored in it. However if it can’t find a running agent it will start one for you.

We still recommend that you have an ssh-agent running before entering the chroot. See [https://wiki.archlinux.org/index.php/SSH_keys#SSH_agents here] for more info on ssh-agents.

====GPG Key And Agent====

For generation of package only key pair, please refer to [https://wiki.archlinux.org/index.php/GnuPG Archwiki’s article about GnuPG] for details.

The buildsystem is designed to use your running gpg-agent and query that agent for passphrases.

=====Activation=====
gpg-agent can be set to automatically start when the socket is used.

 $ systemctl --user start gpg-agent.socket

Tip| You can make it permanent by enabling the systemd units:
  $ systemctl --user enable gpg-agent.socket

Since GnuPG 2.1 the sockets are created under $XDG_RUNTIME_DIR/gnupg/ aka. /run/user/$UID/gnupg by default.

Note|If you set <code>GNUPGHOME</code> the sockets will be located there. You can always find out where your sockets are, by running:
gpgconf --list-dirs

If you need the agent to start right now:

gpg-connect-agent /bye

=====Working With Graphical UI=====

To get a working graphical signing mechanism, you need to configure GPG to use a graphical pin-entry program instead of the default curses-based program. Edit the following file so that pinentry-program points to a GUI pin-entry program such as /usr/bin/pinentry-qt:
File|name=~/.gnupg/gpg-agent.conf|content=pinentry-program /usr/bin/pinentry-qt
keep-display

Note| This is the default value for Chakra
Tip| you can use gpg-agent acting as ssh-agent for graphical passphrase management. For further reading on this topic, please refer to [https://wiki.archlinux.org/index.php/GnuPG#SSH_agent| Archwiki's article].

===Configuration For Local Build System===

Maintaining a package locally is much easier than doing so on remote server.

The network involved: host→container→remote.

The steps are:

  • On host system: [[#Activation| start]] gpg-agent socket listener.
  • In chroot: [[#Retrieve your public gpg-key |import]] your public key

=====Validation=====

You can check the status with:
$ systemctl --user status gpg-agent.socket
Both should be green Active: active (listening) or Active: active (running)

During enter_chroot.sh, there should have an indication:
:: found main socket on local system

And you can try to build any small package and see if it signs.

If package signing fails, check the steps above before entering the buildsystem.

===Configuration For Build System On Remote Server===

There are two ways to be able to sign on remote server:

  • copy your secret key on to a trusted server, and do the packages as is locally (though with only terminal prompt).
  • if you do not fully trust the server host, you should only pass the public key and the agents, and you would be prompted with passphrase request locally.

We would introduce the second condition only, as the first one is the same as above.

The network involved: host→build server→container→remote.

====Start GPG Agent Before Entering the Chroot====
Please refer to [[#Activation| above topic]] for gpg-agent activation.

====Forwarding GPG-agent over SSH====
Note|This requires OpenSSH >6.7 and GnuPG >2.1.

First, you’ll need a ssh key, and a gpg key pair on your host.

=====On Host Machine=====
Edit your gpg-agent.conf to ensure popup from your desktop File|name=~/.gnupg/gpg-agent.conf|content=keep-display

Restart gpg-agent:
$ systemctl --user restart gpg-agent.socket

======Login to remote server======
Use this script to login to remote server and forward your gpg-agent. You can store it in your home directory on the buildserver.
File|name=ssh-gpg|content=
#!/usr/bin/env bash

remote_socket="$(ssh “$@” gpgconf --list-dirs agent-socket)"
local_socket="$(gpgconf --list-dirs agent-extra-socket)"
[[ ! -S “$local_socket” ]] && local_socket="$(gpgconf --list-dirs agent-socket)"
[[ ! -S “$local_socket” ]] && gpg-connect-agent /bye

ssh -R"$remote_socket":"$local_socket" “$@”

Before running it, make sure it is executable:
$ chmod +x ssh-gpg

Usage:
$ ssh-gpg -p

Warning|You must not see this line
ic|Warning: remote port forwarding failed for listen path /path/to/S.gpg-agent
Note|It is suggested not to use your main socket on remote server, consider the nature of security risk, thus you may want to prompt an extra socket for remote usage

 $ systemctl --user start gpg-agent-extra.socket

retrieve the socket path with gpgconf --list-dir for instance can be:
extra-socket /run/user//gnupg/S.gpg-agent.extra
*Restart gpg-agent
$ systemctl --user restart gpg-agent.socket
*the ssh argument becomes like this: -R $(gpgconf --list-dir socketdir)/S.gpg-agent:/path/to/socket

Tip|You can also store common argument in ~/.ssh/config, see <code>man ssh_config</code> for more detail

=====Over Remote Server=====

Edit or create the file ~/.bash_logout so that it cleanly removed gpg-agent and socket:
File|name=$HOME/.bash_logout|content=rm $XDG_RUNTIME_DIR/gnupg/S.

=====Validation=====

During enter_chroot.sh, there should have an indication:

:: found /run/user/<your_uid>/gnupg/S.gpg-agent.extra, will bind it to container!
:: found /home//.gnupg, will bind it to container!
:: found /home//.ssh, will bind it to container!

Now, you should be inside the chroot, the following steps are all done inside your newly created chroot.
=====Inside Chroot=====

======Retrieve your public gpg-key======
$ gpg --import /usr/share/chakra/signatures/<your_id>.asc
or
$ gpg --keyserver --recv-keys <your_pub_gpg_id>

And now, you’re ready to sign packages remotely.

===Manipulating Packages===

====Commit and push the changes====

After [[#Test the Package| testing the package]] you can upload your changes to the PKGBUILD or other files to Git.

[[#Enter a Chroot|Enter the chroot]] again, and run:

$ git status # Just to make sure the changes are correct.
$ git add <package_name> # Where <package_name> is the name of the folder containing the package.
$ git commit -m “package_name: short description of the changes you performed” # Commit the changes

Here are some examples of what a git commit message could include:

  • vlc: 2.0.2 # When updating to a new version

  • blender: rebuilt against python3 # When doing a group upgrade

  • firefox: added patch for KDE integration # When adding a new patch

    Warning| Add one package/folder per commit, multiple folder in a commit is only allowed during [[#Package Chains| package chain]] upgrade.
    Note| You can find more info about git [http://www.git-scm.com/book/en/v2 here].

$ git push # This is what actually send the changes to the Git repository.

{| style=“float: right; text-align: center;” border=1px cellspacing=0
|gp
is alias for
git push
|}
Tip| Despite the one-per-commitment limitation, you can push multiple commitments only once after multiple git commit

Warning| Among doing following direct manipulation with binary files, please verify that things are signed successfully.

For now, we only indicates whether it pass or not, we’ll implement exit with error in the near future.

  • A typical database success would be display as:

downloading and signing the database DONE
building file list …
4 files to consider
(…)

  • A typical package success would be:

==> Signing package…
-> Created signature file /chakra/_repo//<full_pkg_file_name>.pkg.tar.xz.sig.

  • A typical package upload would be:

:: You are about to upload (1) packages and (1) signatures to «repo-x86_64»:
/chakra/_repo//<full_pkg_file_name>
/chakra/_repo//<full_pkg_file_name>.sig

====Upload the Package====
After that you can upload your new package(s) with:
upload.sh

You can use the -r switch to define a specific repository to which to upload the package:
:upload.sh -r repository_name ‘’’’‘package_name’’’’’
The -r switch is agnostic of the order and the command can include more than one package. As a result one can also use:
:upload.sh ‘’’’‘package_name’’’’’ ‘’’’‘package_name’’’’’ -r repository_name
and
:upload.sh ‘’’’‘package_name’’’’’ -r repository_name ‘’’’‘package_name’’’’’

====Update an Existing Package====
First thing you need to do is to update the Filename|PKGBUILD of the application you are updating. So:
*Change the pkgver variable to its new version, or increase the pkgrel variable if you are rebuilding the same version of the package (i.e. with an additional patch).
*Update the md5sum (if sources changed). You need to run makepkg -g >> PKGBUILD from inside the folder that contains the PKGBUILD to do that, and should then move the new md5sum to its right position, after the source variable.

Then you can follow the steps to [[#Add a New Package|add a new package]], just skipping the first one (build package creation).

====Delete a package====

To delete a package or group of packages from the current chroot repository name, run:
:remove.sh ‘’’’‘package_name’’’’’
or from a different repository, run:
:remove.sh ‘’’’‘package_name’’’’’ ‘’’’‘repository’’’’’

====Move a package====
To move a package or group of packages from a repository to another, run:
:move.sh ‘’’’‘package_name’’’’’ ‘’’’‘source_repo’’’’’ ‘’’’‘target_repo’’’’’
‘’’’‘target_repo’’’’’ is optional, the script will try to detect the target repository.
:move.sh ‘’’’‘package_name’’’’’
:move.sh ‘’’’‘package_name’’’’’ ‘’’’‘source_repo’’’’’

2 Likes

see also: same topic with better readibilty