Log in

No account? Create an account
Environment and Stacks Working Group discussed again what our group should
be doing. I guess this time we broadly defined first goal. At least it
looked like consensus of opinions.

bkabrda proposed topic:
tools for setting up development environments/more
automation for packaging/providing stacks. This topic was discussed for
rest of the meeting.

pingou joined our discussion, because he had work on a tool, which could
help with automation of package reviews outside of bugzilla. He might be
able to work on it more, after he will finish pkgdb2.
The tool can work with packages like R, perl, python and php. All these
packages have *2spec or *2rpm tools, which already can help with packaging.
Haskell can be build from upstream into srpms with cabal-rpm. The
critical part of reviews are licenses. With cooperation with copr we could
have automatically update repositories generated from upstream sources.
tjanez said, it could be incubated within our WG first and then proposed
to general (core) Fedora audience. In my opinion we could put those
packages into non Fedora repo for a while and decide the Fedora workflow later.

These sub-goals were summarized mainly by tjanez:
To provide repositories with automatically updates specs and generated rpms of
Python, Perl, etc. modules that are ALREADY in Fedora.
* In a way, the repository would follow the upstream release schedule.
* As soon as the upstream puts it on CPAN, PyPI, it would be available in the
repo unless there is a major version with API/ABI bump that we cannot back port.
* ABI/API breakage should be checked by api-checker. Tools exist, just no-ones
runs them automatically on new packages.
* The automatic packaging helps to see the automatically updated spec file and
the generated rpm, so they have less work updating the packages. Eager users
can enable them AS IS.
* The other idea is to enable easier/quicker packaging of dependent RPM files
by generating spec files for the packager automatically by review tool.
* There was also an idea regarding trying out a new kind of package review
process (via to-be-developed review app). There already exist some rebase
helper (hhorak will find out more).
* Some sort of CI (common integration) for our package repositories.
* Jenkins http://jenkins.cloud.fedoraproject.org/job/javapackages-tools/
(that's CI for Java tooling - requires/provides generators etc).
* Pingou had started to work on something using datagrepper that we could use
to rebuild automatically all packages that have not been rebuild for 1 Fedora.

See full log for whole conversation.


Developer Conference - February 23

I was speaking about Software Collections on Developer Conference in Brno. I forgot to mention that we have now new Red Hat developers blog, which contains howtos for developers on RHEL. People interested in collections can contact us on our mailing list.

Tom 'spot' Callaway presented his vision of new tools, which can help users with talking to developers. For example he and his team are working on Hyperkitty, which will be a plugin for mailman. The output would look like a forum where you can give karma to users or posts. By default, only the high karma posts will be shown, something like slashdot comments, so people won't be bored by trolling so often. He's also focusing on user experience on GNOME3. Based on his data he proposed improvements for tools like PackageKit. I guess such improvements could be helpful for Fedora community.

Leslie Hawthorn was speaking about Negotiation theory in IT environment. I heard similar talks on various soft skills trainings, but Leslie is a great speaker and she made people laugh.

Vratislav Podzimek tried to talk us into writing 3rd party plugins for anaconda. After a complete rewrite of anaconda many people were upset by the current UI, but the big changes were done inside of the code. Anaconda was rewritten to be more modular. Vratislav with Chris Lumens held a public discussion about new anaconda, which sounds to me as a good step forward to community and rest of the system, which is depedent on anaconda (firstboot, initial experience,...).

Adrian Schroeter presented the Open Build Service, which is used by SUSE, but it can also create builds for various distributions. OBS is automagically creating rpm, deb, ... from sources. They are using 'projects' for builds, so they are able to quite easily rebuild bunch of packages marked as a  one project.

The most important on all conferences is speaking to people. I met many of them on the evening social event. This year, Fleda (a musical club) was rented with a life band and DJ in the late evening.

semi-automatic generation of collections

Slavek wrote nifty utility called rpm2scl. It's not packaged into Fedora yet, but you can download it. I test it on one of my  (Perl) package.

# easy_install rpm2scl

rpm2scl  perl-Date-Manip.spec > ~/collection/perl-Date-Manip.spec

--- perl-Date-Manip.spec        2012-07-20 19:52:21.000000000 +0200
+++ perl-Date-Manip2.spec       2012-07-20 19:52:27.000000000 +0200
@@ -1,4 +1,7 @@
-Name:           perl-Date-Manip
+%{?scl:%scl_package Name:           perl-Date-Manip}
+%{!?scl:%global pkg_name %{name}}
+Name:           %{?scl_prefix}perl-Date-Manip
 Version:        6.30
 Release:        1%{?dist}
 Summary:        Date manipulation routines
@@ -8,24 +11,24 @@
 Source0:        http://www.cpan.org/authors/id/S/SB/SBECK/Date-Manip-%{version}.tar.gz
 BuildArch:      noarch
-BuildRequires:  perl(Carp)
-BuildRequires:  perl(Encode)
-BuildRequires:  perl(Exporter)
-BuildRequires:  perl(IO::File)
-BuildRequires:  perl(Module::Build) >= 0.3600
-BuildRequires:  perl(Storable)
-BuildRequires:  perl(Test::More)
-BuildRequires:  perl(YAML::Syck)
+BuildRequires:  %{?scl_prefix}perl(Carp)
+BuildRequires:  %{?scl_prefix}perl(Encode)
+BuildRequires:  %{?scl_prefix}perl(Exporter)
+BuildRequires:  %{?scl_prefix}perl(IO::File)
+BuildRequires:  %{?scl_prefix}perl(Module::Build) >= 0.3600
+BuildRequires:  %{?scl_prefix}perl(Storable)
+BuildRequires:  %{?scl_prefix}perl(Test::More)
+BuildRequires:  %{?scl_prefix}perl(YAML::Syck)

-Requires:       perl(:MODULE_COMPAT_%(eval "`%{__perl} -V:version`"; echo $version))
+Requires:       %{?scl_prefix}perl(:MODULE_COMPAT_%(eval "`%{__perl} -V:%{?scl_prefix}version`"; echo $version))
 # This package was formerly known as perl-DateManip
-Provides: perl-DateManip = %{version}-%{release}
-Obsoletes: perl-DateManip < 5.48-1
+Provides: %{?scl_prefix}perl-DateManip = %{version}-%{release}
+Obsoletes: %{?scl_prefix}perl-DateManip < 5.48-1
It does need some tuning for Perl:
+%{?scl:scl enable %scl - << \EOF}   
-%{__perl} Build.PL installdirs=vendor
+perl Build.PL installdirs=vendor
+%{?scl: EOF}

+%{?scl:scl enable %scl - << \EOF}
./Build test
+%{?scl: EOF}

The utility is written in python, so no patches from me. I guess for python and ruby gems will be rpm2scl working like a charm.

current state of Software Collections

From the first time, when I posted testing repositories for Perl and Ruby, happened a lot. First product using the Software Collections - Development Toolset has been released (see press release). The product supports gcc-4.7.1 and other developer tools for RHEL-6.

Various packages for EL-6 branches can be found on the Software Collections home page. Perl version is still 5.14.2, I'm planning 5.16.0 as soon as it's rebuilt in rawhide. Ruby was updated to 1.9.3 with Rails 3.2.3 and it's probably the most tested collection. For running any of these collections, you need to add a repository. In this case:
$ wget http://people.redhat.com/bkabrda/scl_ruby193.repo
enable the repository or run
# yum --enablerepo=scl_ruby193 install ruby193
and you can start playing with the latest Ruby for example in new shell:
# scl enable ruby193 bash
Other collections offer python3.2, httpd2.4, mysql5.5, postgresql9.1, or unixODBC2.3.1. All of them are built for EL-6 branch, some of them have testing builds for F-17. The main use cases and general packaging advice are documented very well and we are trying to keep them updated. I received few emails with questions about building your own collections. Therefore I started a public mailing list , where you can share your ideas. trac can be used for reporting scl related bugs.

Developer Conference - day 2

Lukáš Czerner had a one of the best talk here. Benefits of btrfs are a scalability more than 16EiB (ext4 can reach 16TB). It should be possible to do a very fast file system checking, incremental backups and snapshots. It should be easy add/remove drives or add subvolumes. The main deficiencies at the moment are: fsync (not fully functional), virtualization is quite slow and also encryption is not implemented. It sounds to me like not ready as default for Fedora, but it might be good choice in the future.
Later that day presented Edward "Joe" Thornber and Zdeněk Kabeláč Thin provisioning and snapshots in device mapper. Joe showed the theoretical part of thin provisioning and Kabi showed how does it work in few easy commands. Lvm commands are often seen as too complex and hard to use by an average user, but the howto showed it doesn't have to be true.

Bryn Reeves had his second talk - How to lose data and implicate people, which was the best talk from the whole conference.
He made a summary of common scenarios how to lose data. His presentation skills are awesome, so better look on the video, but the most funniest quotes were:
  • How long is an ohnosecond? Elapsed time between making a catastrophic error and realising what happened.  
  • Backups? We've heard of them...
Milan Brož had a talk about the disk encryption. He mentioned the benefits of a full disk encryption, where the key removal also means easy data disposal. Milan listed the possibilites of full disk encryption: hardware - encryption on disk controller (usb2sata on disk) - if the board broke, it's hard to save data. And the software encryption: truecrypt,  loop-AES with external store for key, bitlocker (proprietary, windows), luks/dm-crypt - based on a strict separation of the disk encryption engine (dm-crypt) and the key-management (LUKS).
Don't forget to vote for your favourites.

Developer Conference - day 1

Radovan started the event with a short speech and also announced the Open House in Red Hat Brno office on April 4th as the next Red Hat related event.
František Řezníček from MRG (Messaging, Realtime, Grid) spoke after him. His talk was called Towards Unified Messaging, where he presented their work, mentioned protocols used in their project - qpid and amqp. He shortly explained the asynchronous messaging and how the cooperation works between server and users.

Bryn Reeves from support of storage had a lovely talk called - Supporting the Open Source Enterprise. He mentioned few examples from his area of work, but it's probably the same for all open source projects. Problems with support of a project in various branches, support of older releases, which are unsupported by upstream and so on. What has changed in last few years are tools. Git made things easier, but there are also other tools worth looking at like OpenGrok or cscope.

Jan Hutař presented "Software Robot Competition around the world". He made an overview of different competitions in different disciplines and showed the most funniest videos. For my colleagues was important the Red Bot competition based on FIbot, where he announced the winner of the internal competition. Jan promissed the best videos from the Red Bot on youtube.

Stanislav Kozina spoke about What can Linux learn from others. I felt that this talk was about how cool is Solaris, namely zfs and dtrace. I heard many times praise of both and they probably have many cool features. But I would prefer a comparison of programs or what are the deficiencies of these two. I guess drawbacks are everywhere.Phil
Phil Knirsch made an overview of rpm and yum. I assume they spent a lot of time on looking into "yum is so slow" issues and aimed their work on the most irritating bugs. According to their research is rpm+yum only 4x slower than untar, so it's very good because rpm+yum are doing installation and scriptlets, SElinux, depsolving and many more. Some projects claimed to be faster, but in that case they are "cheating", because they are skipping some work. The most important future plan is using libsolv by SUSE as the depsolver. On the top of depsolver will be a new project, which will merge rpm+yum functionality into one tool, which will be introduced in Fedora 18.

Jarda Škarvada presented news from the Power Management. He announced the test day, where anyone can bring his/her own laptop into Red Hat Brno office for measurement or do the testing at home. They introduced new release of tuned 2.0, which have new profiles for power saving. He proposed a little controversial change to stop desktop application to run their flash/javascript/whatever during their minimalization. It would require patching for all Desktop Environments.

Harald Hoyer and Kai Sievers presented their new look on FHS called - A streamlined and fully compatible Linux Filesystem Hierarchy. IMHO it's another change for a change, which will probably create more problems for Fedora maintainers. I'm not quite sure what this change should bring to the user or the developer. Audience was arguing and they were arguing with Harald and Kai. We spoke afterwards, but I can't say I'm convinced...

ruby 1.9.3 in EL-6 branch

The Ruby repository contains ruby-1.9.3, rails-3.0.10, and group of essential gems. 
Everything from collection can be installed by:
yum install stack_rails_3.0

Now, when the collection is installed, the rails version can be verified.
stack enable rails_3.0 'rails -v'

As mentioned above, the only difference between running commands inside and 
outside stack is using the stack command. If you want run your own rails3 
application, then you simply install the Ruby stack. In the directory, 
where is located your application run:
stack enable rails_3.0 'rails new app_name'

Now you can develop in this directory, but everything must be executed within 
the stack environment. For running rails server you can use the command:
stack enable rails_3.0 'rails server'
But you can also run a new shell:
2/ stack enable rails_3.0 bash
where the Stack Ruby will override the system Ruby. In the new shell will be used
the Ruby from Stack, but also all gems and everything, what is provided by this
software collection.

Notes to installation: The Ruby software collection was built with the old utility and macros, so it's needed to install stack instead of scl-utils from EPEL before you start. In the future will be all software collections using dsc prefix and scl-utils from EPEL repository.

Update: The main package was renamed to scl-utils and repositories are also stored on different places. See status of Software Collections for more information or join our mailing list softwarecollection.


Dynamic Software Collections

Collections should provide new or old version of software packaged in rpm. Imagine you have for example EL-6 and you want use new features from Perl 5.14.x. Update of system Perl is not possible, because it would bring changes incompatible with other system packages. Or you have application running on Ruby 1.8.x and in Fedora 17 will be Ruby 1.9. Collections are groups of packages, which are using different paths for installation, so those packages shouldn't influence the system environment. It should be possible work with both versions. The system one is linked to application like vim or netsnmp and the collection package could be executed in his own shell and used only there.

Currently, there are prepared few repositories of Ruby and one repository of Perl for testing purposes.

As the maintainer of Perl I'll show all examples on perl 5.14.2, which is prepared for EL-6.

How does it work:
 * install scl-utils. This package will provide macros, which are needed for running software collection. The scl-utils-build package is needed only as build requirement.
 * pick a collection, in this case Perl. yum install perl514* Perl collection includes upstream tar ball, DBI, FCGI.

The installed version can be verified by:
scl enable perl514 'perl -V'

Setting SElinux rules should be easy by using same contexts on new directories:
semanage fcontext -a -e /usr /opt/rh/perl514/
restorecon -R -v /opt/rh/perl514/

If you run new session, you can use there new version only:
scl enable perl514 bash

For example execute perl -V, which shows information about your Perl. Let's look at paths specific for this collection:


Ruby collection is using rails_3.0 instead of perl514 in path. They started creating their collection sooner than me, so they are using old macros. For working with their ruby stack, you need install macros and enable the repository for Fedora or EL-6.
yum install stack_rails_3.0 should install whole Ruby stack and stack enable rails_3.0 'rails -v' should verify the path.

Update: Next article should be about the functional installation of Ruby 1.9.x on EL-6.

Fedora elections

Did you know that Fedora elections already started? End of voting is 8th June for FESCo and 9th for Board. Don't forget to vote for your favourites!

Howto create RPM filters

Rpm 4.9 brought change of macros. They do not have much documentation, but I was able to create something with help of rpm guys. Thanks.

Now you can define and use your own generator of dependency, which could be handy if your packages have specific needs. New macros have only extended regular expressions, not Perl regular expressions :(

1. create filter for your favourite package:

2. define your own generator
%global __favourite_provides /usr/lib/rpm/favourite.prov
%global __favourite_requires /home/user/favourite_inprocess.req

3. filter *.so, which don't have to be provided
%favourite_default_filter %{expand: \
%global __provides_exclude_from %{_share}/favourite/.*\.so|%{_sbindir}/.*\.so

4. change it to filter *.so and also unwanted require MyPage
%favourite_default_filter %{expand: \
%global __provides_exclude_from %{_share}/favourite/.*\.so|%{_sbindir}/.*\.so
%global __requires_exclude favourite\\\\(MyPage\\\\)

5. use it in your favourite package before prep
This is favourite package..



6. Provides and requires added manually before prep section are not filtered, because they are added after filtering.

And example for perl-DBI package. In case there is more provides to be filtered, it's getting uglier.

# filter provides, which you don't want in all Perl packages
# don't overwrite exclusion of provides, just add another
%global __provides_exclude %{?__provides_exclude}|perl\\(DBI\\)
%global __requires_exclude %{?__requires_exclude}|perl\\(RPC::