mmaslano

how hhorak's idea about summary from each WG worked

Honza Horak (hhorak) worked on summary what every WG is doing and what's the status. It took him a long time, so we've agreed to contact all groups with requirement on sharing the progress. Shortly, cross posting a short summary to all groups wasn't seen as a good idea, but using more #info and #action during meeting could improve readability of meeting logs. Chairperson will do better meeting minutes and put them on blog (shared on fedora.planet) to do them more accessible.

ACTION: hhorak will keep trying to get summary of working groups' discussion (ML, meetings) to other groups members

epel/epic proposal - chat with smoodge
Faster EPEL repository is only proposal right now. Members of Env and Stack should follow epel-devel mailing list for discussion about the proposal.

openfloor
Chairperson for next meeting will be picked at end of the current meeting.

ACTION: hhorak will be chairman and secretary next week

ACTION: mstuchli will work on Fedora Docker build service

• We need to figure out whether we want private Docker registry or not

Koschei is not only the evil guy from Slavic legends, it's also a tool using Koji for Continuous Integration written by Mikołaj Izdebski and Michael Šimáček. Simple rebuild can test whether tests enabled during build time are still passing or at least if build is still passing after change of dependencies. Regular rebuild can improve state of packages, because build failures are usually found at mass rebuild, which is happening less than once a year.
Real continuous integration means rebuild of whole project after every commit and execution of test suite. When we are talking about Fedora, we can't afford it, because complete rebuild would take a few days. Koschei is running scratch-rebuilds on Koji, because it's executed in same environment as Fedora builds. Also Koji has already lot of hardware and it's maintained by infrastructure team. Scratch-rebuilds are done only on set of packages, when load on Koji is low and priority of package is high. Priority is mainly based on:

• time since last rebuild


• dependency changes


• previous state

Koschei is using:

• fedmsg for receiving information about new builds or repos


• for analysis of change of dependencies is used hawkey.

You can look at the list of rebuild packages and what triggered the rebuild. The source can be found at the github. Good place for discussion would be probably Env and Stack Working Group mailing list: https://lists.fedoraproject.org/pipermail/env-and-stacks/2014-June/000428.html

Fedora::Rebuild tool is a project old few years. It evolved from first scripts to proper tool. It's developed by Petr Písař, who started with nifty script for rebuild of set of Perl packages. Usually packages or whole distribution are rebuild according to alphabetical order of packages. This approach doesn't work well when you have huge number of packages with dependency cycles and you need to boostrap it. The tool was developed for Perl, but it might be used for other defined sets of packages. If you've tried on a group of packages or even whole distribution, let us know how it worked.

How rebuild with bootstraping works
At least rebuild of Perl packages is done annually with new release of Perl, which might have some incompatibilities. Every Perl related package should contain a macro, which says which version was used for build of the package.

Requires:       perl(:MODULE_COMPAT_%(eval "`%{__perl} -V:version`"; echo $version))

All those packages are marked for rebuild (~2500 in Fedora). Firstly is Fedora::Rebuild tool counting dependencies (few minutes). Surprisingly, there doesn't exist any tool, which provide similar feature. After dependencies are solved, only packages with satisfied dependencies are built. Usually, only main perl package and perl-srpm-macros can be rebuild in first round, because they don't depend on anything else.
Even after few rounds, the rebuild can't continue because of unresolvable cyclic dependencies. Cycles can be broken by bootstraping macros in specfiles. The bootstrap macro is defined in perl-srpm-macros, which is always tagged into the buildroot. Breaking cycles is lot of manual work, the best way how to do it is look on visualisation and try to break them. When everything had passed, then is needed to undefine macro in perl-srpm-macros and run rebuild again. Everything should pass in one final round.

Example of breaking cycles for rebuild:

%if !%{defined perl_bootstrap}
BuildRequires: perl(File::Find::Rule) >= 0.32
BuildRequires: perl(Test::More) >= 0.47
BuildRequires: perl(Test::Script) >= 1.03
%endif

%check
%if !%{defined perl_bootstrap}
make test
%endif

Set up Fedora::Rebuild

Rebuild has options, which can be set in /usr/bin/rebuildperl or you can create copy of this file or copy the configuration only as ./rebuildperlrc and modify. There are some known bugs like dist-git is taking only 10 connections, so it's not wise to set threads to more than 10. It's not very user friendly nor documented, but at least documentation is in todo.

my %config = (
# lists of packages for rebuild, list of failed ...
all => 'all',
    done => 'done',
    failed => 'failed',
# define work directory
workdir => 'workdir',
    repodir => 'testrepo',
    repourls => [
        'http://lab.com/pub/fedora/linux/development/rawhide/x86_64/os/' ],
# for mirrored repos, simple 'build' is for direct access to koji repos
mock_install_group => 'buildsys-build',
    architecture => 'x86_64',
    anonymous => 0,
    dist => 'rawhide',
    target => 'rawhide',
    message => 'Perl mass rebuild',
# restrict to list of dependencies
buildrequiresfilter => sub { $_[0] !~ /^\s*perl\(/ },
# rebuild can run in koji, mock or on local
mode => 'koji',
    threads => 16,
    loadthreads => 4,
    selectthreads => 1,
# how many failures must happened before rebuild stop
failurelimit => 8,
    ordered => 1,
    verbosesolver => 1
);

Build

Rebuild of Perl packages in Fedora takes about 4 days of quad core cpu. Good luck with compiling!

Some time ago was released RHSCL-1.1. At this point we had more collections with more dependencies and we were looking for something like shared group of packages among collections. For example there are Ruby&Rails collection and it might need node.js for processing javascripts or mongodb can use v8 for their work. The v8 was needed by more collections, it didn't make any sense to "bundle" it into all of them. The v8 collection was created as separated thing and other collections could use it. There was slight risk that node.js might want newer version of v8 soon, but still possible maintenance of two v8 versions looked better than start bundling v8 into three collections right now.

There exists one v8 collection called v8314. Rpmbuild must be aware about existence of some v8 macros, but v8314-build can't be installed, because then all prefixes of mongodb would be also v8. The build sub-package was split into two parts and it's highly recommended to create -build and -scldevel for all new meta-packages.

The build sub-package contains just:
cat /etc/rpm/macros.v8314-config
%scl v8314
%scl_vendor rh

The scldevel sub-package contain macros, which are needed also for build of mongodb, when looking for v8 during build:
cat /etc/rpm/macros.v8-scldevel
%scl_v8 v8314
%scl_prefix_v8 v8314-

Okay, now just define v8314 in mongodb24 specfile and you can build it with support of v8. The solution is far from perfect, but it works.

%{!?scl_name_base:%global scl_name_base mongodb}
%{!?scl_name_version:%global scl_name_version 24}
# particularly useful for mock
%{!?scl:%global scl %{scl_name_base}%{scl_name_version}}
%scl_package %scl
# define version of v8, we want to use. Requires cant' be defined as %{?scl_v8_%{scl_name_base}}, because it would append mongodb.
%global scl_v8 v8314
%{?scl:%global scl_v8_prefix %{scl_v8}-}
# do not produce empty debuginfo package
%global debug_package %{nil}

Summary: Package that installs %scl
Name: %scl_name
Version: 1.1
Release: 2%{?dist}
Requires: %{scl_v8}
Requires: %{scl_prefix}mongodb-server
BuildRequires: scl-utils-build, help2man
...
%package runtime
Summary: Package that handles %scl Software Collection.
Group: Applications/File
Requires: scl-utils
Requires: %{?scl_v8_prefix}runtime
Requires(post): policycoreutils-python, libselinux-utils
...
%install
# enable script needs to source v8 collection defined at the start of this specfile
cat >> %{buildroot}%{_scl_scripts}/enable << EOF
export PATH=%{_bindir}\${PATH:+:\${PATH}}
...
export XDG_DATA_DIRS="%{_datadir}:\${XDG_DATA_DIRS:-/usr/local/share:/usr/share}"
. scl_source enable %{scl_v8}
EOF
#create scldevel
cat >> %{buildroot}%{_root_sysconfdir}/rpm/macros.%{scl_name_base}-scldevel << EOF
%%scl_%{scl_name_base} %{scl}
%%scl_prefix_%{scl_name_base} %{scl_prefix}
%%scl_v8_%{scl_name_base} %{scl_v8}
EOF
...

The scl_source macro will automatically enable v8, when you enable mongodb, because it's hard requirement of mongodb. Otherwise users would have to enable both collections manually. In scldevel is needed only definition of name of v8 collection.

On Copr are currently so many projects and it's boring to find a webpage of interesting project, download a repository, put into /etc/yum.repos.d/... Mirek wrote a nifty plugin for dnf, which can make it much easier.

1. are you on Fedora 20, right?


2. you need to install dnf and plugins

[root@localhost ~]# yum install -y dnf dnf-plugins-core

3. list users Copr projects

[root@localhost ~]# dnf copr list dvratil dvratil/plasma-next : Repository with unstable builds of Plasma Next
dvratil/kde-frameworks-nightly : No description given
dvratil/kde-frameworks : Unofficial builds of KDE Frameworks 5 and Plasma 2. Not for daily use. dvratil/kde-frameworks-upgrade : No description given


4. enable some cool Copr repository, don't forget to define architecture. It fails without it (for now).

[root@localhost ~]# dnf copr enable dvratil/plasma-next fedora-20-x86_64
You are going to enable Copr repository.
Please note that this repository is not part of Fedora distribution and may have various quality. Fedora distribution have no power over this repository and can not enforce some quality or security level. Please do not file bug reports about this packages in Fedora Bugzilla. In case of problems you should contact owner of this repository.
Do you want to continue? [y/N]: y
Repository successfully enabled.


5. Try to install something

[root@localhost ~]# dnf install kde5-plasma-nm ...

We spoke many many weeks about it and taddaaa!

There is almost finished plan how to add Playground repository, which can contain various stuff. From chromium (bundling) to interesting packages, which didn't make it into Fedora yet. People will be using Copr for build of those packages and repositories can be nicely handled by Copr plugin for dnf.
https://fedoraproject.org/wiki/Env_and_Stacks/Changes_Drafts/Playground_repository

Software Collections even for Fedora. The guidelines are not finished yet, but let's hope for the best. I'd like to create at least one collection of ruby193, which was most wanted.
https://fedoraproject.org/wiki/Env_and_Stacks/Changes_Drafts/SCL

After discussion with pingou about some parts of PRD, he proposed Automated packages review tool. If everything goes fine, reviews can be done on server outside of bugzilla, which is really not a good tool for reviews. It's a big project planned for Fedora 22.
https://fedoraproject.org/wiki/Env_and_Stacks/Changes_Drafts/Automated_packages_review_tools

whole January, mid February
PRD was approved with few comments, which were included into last version of PRD. Debi Rieden became a voting member, because previous member was missing on meetings and email last few weeks.

Wiki pages for the Working Group were finally created https://fedoraproject.org/wiki/Env_and_Stacks. Now is Env and Stack Working Group thinking about tasks planned for Fedora.next. We started with requirements for additional repository. On DevConf was additional repository mentioned few times, so it looked like good start, but no one from other Working Groups stated their use-cases yet [1]. We would be really great for input, otherwise we have to think about wokflow without real use-cases. Nevertheless members of Env Working Group started to think about their own use-cases, so it's a start [2].
People from Cloud and Server Working Group mentioned to me they intend to use SCL in their products, so we should probably think about workflow for SCLs as soon as they are fully approved by FPC.

[1] https://lists.fedoraproject.org/pipermail/devel/2014-February/195483.html
[2] https://lists.fedoraproject.org/pipermail/env-and-stacks/2014-February/000173.html

I looked at Mirek's screenshots tutorial and start building perl516 collection. I already had srpms and rpms from Jitka, who built it locally, so it should be just a rebuild in copr. I wanted to try copr with huge number of packages.

I started with a build of the meta-package - perl516. It was needed to add into a buildroot of copr the scl-utils-build package. After the first package was built, a repository containing the meta-package was build. I've added this repository as additional repository, therefore the perl516 meta-package was available for build of the collection. For build of whole collection were also needed perl516, perl516-build and scl-utils. Then I was able to rebuild perl516-perl, the core perl package.



Rebuild of perl takes a lot of time, because there are cyclic dependencies and it has to be bootstraped for new version. But I decided to use rpms as build requires from local rebuild of perl516. It might be a problem if the local build was done on different architecture, but this wasn't the case.


It is possible to add more srpm into New Build, so I did. After few beginners mistakes I was able to run complete rebuild. The repository can be found on the main page of coprs or here.

17th December
Last week I spoke with a guy from Big Data SIG. I invited him for WG meeting, because his SIG has troubles within current Fedora workflow. Big Data SIG spoke about their issues with maintenance of languages, which is not their primary goal. They are using mainly Scala, but also some Java packages and node.js.
Big Data specific case is right now, they want to use Scala for real work on Fedora, but they have to either maintain it or install from upstream. They'd like to see some improvement for inclusion of new languages/framework, which could speed up development. tjanez promised to sum up needs of Big Data SIG from meeting minutes.

I started work on PRD. On one of previous meetings people tend to like Cloud PRD, so I used it as a template and prepared a draft. pkovar promised to fix my English and add into tasks his plans for documentation. Every member of WG should work on Tasks, because it's not much time for a preparation of PRD.

Full log

3. December
After another long discussion what we should be really doing (PRD, tasks,...), we started listing what can be added into our scope. I'm afraid without input from other WGs we don't have chance to present useful document.
On meeting were mentioned interesting tools, which can be improved:

• Automated Review tools - help with packaging, review of package, improve infrastructure for import of new packages


• DevAssistant - it can be used as a helper for many developer areas


• work on enhancing build systems (copr & koji)


• Taskotron - Automated task system


• scale manpower of infra/qa/releng to add features to tools and deploy new tools

People has to work on outlining the sections of wiki brainstorming for next week.

10. December
It was very short meeting. Petr Kovar promised to send an analysis of what should be done with docs/wiki to env and stacks mailing list. Honza Horak promised to examine tutorials and howtos for webstack and databases and what could be improved in this area.