Mining the Fedora Infrastructure Bus

If you listen to the Fedora Infrastructure Bus, you will hear things like:

fedora-infrastructure:org.fedoraproject.prod.fas.user.create
{u'certificate':
u'LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUVSVENDQTY2Z0F3SUJ
BZ0lCSmpBTkJna3Fo\na2lHOXcwQkFRVUZBRENCb0RFTE1Ba0dBMVV
FQmhNQ1ZWTXgKQ3pBSkJnTlZCQWdUQWs1RE1SQXdE\nZ1lEVlFRSEV3ZFNZV3hs
YVdkb01SY3dGUVlEVlFRS0V3NUdaV1J2Y21FZwpVSEp2YW1WamRE
RVBN\nQTBHQTFVRUN4TUdabVZrYlhObk1ROHdEUVlEVlFRREV3Wm1aV1J0YzJjeER6QU5
CZ05WCkJDa1RC\nbVpsWkcxelp6RW1NQ1FHQ1NxR1NJYjNEU
UVKQVJZWFlXUnRhVzVBWm1Wa2IzSmhjSEp2YW1WamRD\nNXYKY21jd0hoY05NVEl3TnpFMU1
qSXdNRFE1V2hjTk1qSXdOekV6TWpJd01EUTVXakNCMURFT,
u'i': 1, u'timestamp': 1353031870.694725, u'topic':
u'org.fedoraproject.prod.fas.user.create', u'signature':
u'AvVPjUn0qP+W9qYLDkPFDBdmQDq66NJd2TuPfZZYYbwEkCSOmKsbNnGyGyVhM0n4SoqlNWR0KG49
\nz3FSBgHO2Tytnj+ZltMmeg8GNwaf5ys+bAaiL81AOTeG
ALtPD/iYGiXB0vIjZ685IdPMlu9ZZZej\nauUcUnKuacpzcy17/NA=\n',
u'msg': {u'user': {u'username': u'odemia'}, u'agent': {u'username':
u'odemia'}}}

which in plain text means that a new FAS username was created which goes by the name ‘odemia’.

Or, for example:

fedora-infrastructure:org.fedoraproject.prod.bodhi.update.comment
{u'certificate': u'LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUVTVENDQT
dLZ0F3SUJBZ0lCRnpBTkJna3Fo\na2lHOXcwQkFRVUZBRENCb0RFTE1Ba0dBMVVFQmhNQ1
ZWTXgKQ3pBSkJnTlZCQWdUQWs1RE1SQXdE\nZ1lEVlFRSEV3ZFNZV3hsYVdkb01SY3dGUVl
EVlFRS0V3NUdaV1J2Y21FZwpVSEp2YW1WamRERVBN\nQTBHQTFVRUN4TUdabVZrYlhObk1RO
HdEUVlEVlFRREV3Wm1aV1J0YzJjeER6QU5CZ05WCkJDa1RC\nbVpsWkcxelp6RW1NQ1FHQ1Nx
R1NJYjNEUUVKQVJZWFlXUnRhVzVBWm1Wa2IzSmhjSEp2YW1WamRD\nNXYKY21jd0hoY05NVEl
1Evcit2TFFZZWoxN1cwCi0tLS0tRU5EIENFUlRJRklDQVRF\nLS0tLS0K\n', u'i': 1, 
u'timestamp': 1353038721.092251, 
u'topic': u'org.fedoraproject.prod.bodhi.update.comment', u'signature': 
u'kfsHMnH0zHp6t11woX9uH34t3qDGOtzRHHgrIuUMaBlB8XbNfXl2lQOzhR5iQd3iUIMUh71Q0UPL\njjaIvRJWxL/yz75oiq
btBMYUabi3X0mOmz3dj0rzGeIJVrwVdz+u5ePeBhR6rTDAvrmknrw6lgnr\nyVG7eFk6mQUGXk+MM9o=\n', u'msg':
{u'comment': {u'group': u'proventesters', u'author': u'adamwill', u'text': u'mostly working okay here', 
u'karma': 1, u'anonymous': False, u'timestamp': 1353038719.0, u'update_title': u'anaconda-18.29-1.fc18'}, 
u'agent': u'adamwill'}}

which corresponds to adamwill’s comment on bodhi [1].

If you start storing these messages emitted, you can mine for some interesting statistics. (I am storing all the messages in a file ‘messages’ using a particular format. See the Python source code at the end)

For example, in the last 20 hours or so, 16 new users registered with FAS:

$ cat messages | grep 'fedora-infrastructure:org.fedoraproject.prod.fas.user.create' | wc -l
16

And 175 edits was made to the various wiki pages:

$ cat messages | grep fedora-infrastructure:org.fedoraproject.prod.wiki.article.edit | wc -l
175

You can also look for rawhide and branch compose tree messages:

$ cat messages | grep fedora-infrastructure:org.fedoraproject.prod.compose
fedora-infrastructure:org.fedoraproject.prod.compose.rawhide.start
fedora-infrastructure:org.fedoraproject.prod.compose.rawhide.mash.start
fedora-infrastructure:org.fedoraproject.prod.compose.branched.start
fedora-infrastructure:org.fedoraproject.prod.compose.branched.mash.start
fedora-infrastructure:org.fedoraproject.prod.compose.rawhide.mash.complete
fedora-infrastructure:org.fedoraproject.prod.compose.rawhide.rsync.start
fedora-infrastructure:org.fedoraproject.prod.compose.rawhide.rsync.complete
fedora-infrastructure:org.fedoraproject.prod.compose.rawhide.complete

And as you can imagine, the possibilities are well, endless. Check out the documentation here at [2]. The consumer script I used is at [3]. Thanks to threebean for his help.

[1] https://admin.fedoraproject.org/updates/FEDORA-2012-18105/anaconda-18.29-1.fc18?_csrf_token=482cef87c4c42a4f60ba3faf97bbe16acad4e935
[2] Fedmsg Documentation
[3] Naive Consumer Script

Advertisement

Conflicting libdb dependencies: rpm-libs + httpd 2.2

I ran into a segfault-ing web application while attempting to get Beaker server working on Fedora 17. Beaker uses TurboGears for the web application using mod_wsgi to talk to Apache (httpd-2.2). (I am a little new to the web application world, so ignore any misuse of terms).

The particular use case that triggered the crash was uploading a task RPM to the Beaker server. That was some hint into where to look for the faulting code. The task RPM was being saved fine on the disk, so there was no problem in writing (permissions, etc). So, there had to be a problem in reading the RPM file (which is what Beaker does to retrieve the files and task specifications).

To investigate whether it was a problem reading the RPM file, I simply copied the relevant code from Beaker sources to a standalone Python script. No problems there. Hence, there had to be something wrong with the combination of using the rpm libraries (rpm-libs) together with the web application (httpd).

Debugging httpd with gdb

After futile efforts of inserting exception handling codes combined with logging in the code to bracket the code which was causing the crash, I decided to bite the bullet and just go the gdb way. Turns out it was simpler than I thought. Run ‘httpd’ in single process mode, and  then monitor the error_log file for the wsgi daemon process ID and then attach to it using gdb in another terminal.

Start httpd in single process mode:

# gdb /usr/sbin/httpd
(gdb) run -X

Then attach gdb to the wsgi daemon process (You should ensure via your wsgi configuration to create only one mod_wsgi daemon process with a single thread):

# gdb /usr/sbin/httpd attach 
(gdb) cont.

Once I had done this, I simply performed the action that triggered the crash, and I had a good stack trace, first few lines of which were:

#0 0x0000000100000001 in ?? ()
#1 0x00007fffd7fe1097 in db_init (dbhome=0x7fffdf7a9e70 "/var/lib/rpm", rdb=0x7fffdd052970)
at backend/db3.c:151
#2 dbiOpen (rdb=rdb@entry=0x7fffdd052970, rpmtag=rpmtag@entry=0, dbip=dbip@entry=0x7fffe399ff38,
flags=flags@entry=0) at backend/db3.c:551
#3 0x00007fffd7fe8e53 in rpmdbOpenIndex (db=db@entry=0x7fffdd052970, rpmtag=rpmtag@entry=0, flags=0)
at rpmdb.c:149
#4 0x00007fffd7fe93ef in openDatabase (prefix=, dbpath=dbpath@entry=0x0, dbp=dbp@entry=
0x7fffdf797648, mode=mode@entry=0, perms=perms@entry=420, 
.
.
.

So that pretty much confirmed that it was something that httpd did not like about the rpm-libs which caused it to crash the application. Discussing with Dan, this was indeed a case of conflicting shared libraries and bit more of looking around we found that this was the Berkeley DB database library (libdb) that was the culprit. httpd had both libdb-4.8 and libdb-5.2 loaded in its process maps, also verified with ‘lsof‘ (Thanks to StackOverflow for the lsof tip):

# lsof /lib64/libdb-4.8.so

COMMAND   PID   USER  FD   TYPE DEVICE SIZE/OFF    NODE NAME
httpd    9722 apache mem    REG  253,1  1555128 1189531 /usr/lib64/libdb-4.8.so
httpd   10578 apache mem    REG  253,1  1555128 1189531 /usr/lib64/libdb-4.8.so
httpd   18828 apache mem    REG  253,1  1555128 1189531 /usr/lib64/libdb-4.8.so
httpd   18832 apache mem    REG  253,1  1555128 1189531 /usr/lib64/libdb-4.8.so
gdb     18863   root mem    REG  253,1  1555128 1189531 /usr/lib64/libdb-4.8.so

[root@asaha temp]# lsof /lib64/libdb-5.2.so 
COMMAND    PID   USER  FD   TYPE DEVICE SIZE/OFF    NODE NAME
gdb      18824   root mem    REG  253,1  1756808 1204315 /usr/lib64/libdb-5.2.so
httpd    18832 apache mem    REG  253,1  1756808 1204315 /usr/lib64/libdb-5.2.so
gdb      18863   root mem    REG  253,1  1756808 1204315 /usr/lib64/libdb-5.2.so
qemu-kvm 30866   qemu mem    REG  253,1  1756808 1204315 /usr/lib64/libdb-5.2.so

As you can see, both the libdb versions is mapped to httpd’s process space. Since, httpd itself depends only on libdb-4.8, there is something else in the web application which is bringing in libdb-5.2. That something turned out to be rpm-libs:

# yum deplist rpm-libs |  grep libdb
  dependency: libdb-5.2.so
   provider: libdb.i686 5.2.36-5.fc17

So, that’s the problem and the reason for the crash.

Solution

There is no solution on Fedora 17 at this point of time to this other than trying to get httpd-2.4 which is linked against libdb-5.2.so. However, the Fedora 18 release, currently in development ships with httpd-2.4 which is linked against libdb-5.3.so, same as the rpm-libs version it is shipped with. And indeed, the above crash did not occur there.

Reproducing

To reproduce this with a minimal application, I taught myself how to integrate Flask with mod_wsgi and wrote this simple Flask application, which you can check out here. Follow the steps on the Flask docs for help.

• Sample Flask Application
• Flask + mod_wsgi

Resources

Thanks to Dan on #fedora-qa and Dan in real life who helped me with debugging the crash and thanks to Graham Dumpleton for the mod_wsgi help on #pocoo. Here are some of the docs I referred to:

• Apache HTTPD Debugging
• Debugging mod_wsgi applications

Related bug reports

• httpd+mod_perl and BerkeleyDB incompatibility

self in Python

self keyword is a reference to the invoking class instance/object. Consider the following code:

''' Self really refers to the instance called with '''
class Point:
    
    def __init__(self):
        print 'Self identifier:: ',id(self)

if __name__ == '__main__':
    # create an instance of the class
    p = Point()

    print 'Point object identifier', id(p)
    
    # create an instance of the class
    p = Point()

    print 'Point object identifier', id(p)

As you can see, a class (that does nothing) called Point is defined and then two objects/instances are created. To verify that self is really a reference to the invoking object, we print the identifier for the calling object and self. Unsurprisingly, both are the same.

Update:
Soon after posting this, I realized that self is not a keyword. In fact its just a convention to use self as a reference to the invoking object. You can use anything else, for example, this:

class Point:
    
    def __init__(this):
        print 'This identifier:: ',id(this)

if __name__ == '__main__':
    # create an instance of the class
    p = Point()

    print 'Point object identifier', id(p)
    
    # create an instance of the class
    p = Point()

    print 'Point object identifier', id(p)

GSoC 2012: On-Demand Fedora Build Service: Update #8

I spent some time cleaning up the code, mainly using pylint as an indicator. Also rewrote the fabfile to use fabric commands rather than native Linux commands wherever possible and is cleaner now. The HOWTO is also updated now.

This code refactoring has brought home (quite strongly!) the need for having my unit tests up and running soon. I plan to use py.test for my testing, and hopefully will have some tests ready in a week.

The code is available at https://github.com/amitsaha/gsoc2012_fbs

GSoC 2012: On-Demand Fedora Build Service: Update #7

In my last update, I reported that I had a functional build service, capable of harnessing multiple build nodes and a functional Web UI as well. The process to deploy the build service was however quite manual and tedious. I was looking for a way to help do this without as much manual intervention as possible and fabric (Thanks, Tim) was the answer. I wrote a fabfile for the whole task, called ‘deploy.py’.  As outlined in my earlier post, I am using celery to distribute the build tasks, so the celeryd process needs to be running on the build daemons. I found zdaemon (Thanks Jan on fabric mailing list) to be the easiest way to run the celeryd process as daemons.  The updated code is available in the repository now [1].

There is a HOWTO [2] document, which should help you deploy the service and run your own home based build service. I recently used it to build myself a Fedora Scientific ISO. Please don’t abuse it. There is hardly any error handling now. I am however accepting bug reports now. Thanks for all your help.

[1] https://github.com/amitsaha/gsoc2012_fbs/
[2] https://github.com/amitsaha/gsoc2012_fbs/blob/master/HOWTO

GSoC 2012: On-Demand Fedora Build Service: Update #5

The current github code [1] does quite a few things as of now. Let me try to explain to the changes since last update and my rationale behind them.

Support for building Live images: I attempted to use livemedia-creator (which is going to be THE tool from F18+), but unfortunately ran into issues which prevented me from building images. So for now, I have implemented this feature using ‘livecd-creator’. The Kickstart file (flattened) needs to be specified and other details such as architecture, any extra packages to be pulled from Koji, etc. The specifications are specified via the config/live.conf file.

The User specifications are now completely via .conf files: Myrationale behind that in the first place was that since this code is really going to serve as the ‘backend’, command line arguments could be done away with. But, even if we want to use this as standalone, specifying .conf files is fine as well. (We will see what happens with this after discussing with my mentors). Here is a brief description of the config files in config/

• imagebuild.conf: type of image, architecture, staging area (to be explained later) and email (for notification)
• boot.conf: configuration for boot.iso images
• repoinfo.conf: repository configuration required for the above
• pungi.conf: configuration for DVD images
• live.conf: configuration for live images

The kickstart files if needed are to be placed in the kickstarts/ sub-directory. To use it, you will need to ‘cd’ into ‘image_builder’ directory and run ‘$python run_imagebuild.py’ after setting up the appropriate config files in image_builder/config and kickstart files in kickstarts/
if any.

Support for copying images: I have also now enabled support for copying the images to a ‘staging area’ as mentioned earlier. I assume a passwordless login setup and hence do a ‘scp’ once the desired image has been created. This is how you would use it standalone.

Now, as a first step towards being able to distribute build jobs to different node, I have also now added simple support for carrying out the build process on a different host. This is done by the file
delegate.py.

Here is what it does:

• Assumes that the config/ and kickstart/ files have been correctly setup by the web-form handler or manually.
• Then copies these files to the image_builder/ directory
• It creates a .tar archive of the image_builder
• Then it reads the appropriate node (architecture) from the nodes.conf file and also retrieves the working location specified.
• The .tar file is then ‘scp’-ed to the appropriate location specified
• Then runs the run_imagebuild.py script on the build node by ‘SSHing’
• The image is then automatically transferred to the staging area
  specified as earlier.

To try this feature, simply setup nodes.conf file correctly and the config/ and kickstarts/ in data/ and run $sudo python delegate.py

Note that the specified nodes should have all the dependencies installed, such as lorax, koji (setup correctly), pykickstart and livecd-creator.

[1] https://github.com/amitsaha/gsoc2012_fbs

ownCloud 4 on Fedora 16

ownCloud 4 installation is pretty much similar to ownCloud 3 installation and this shell script here will do the job for you on Fedora. One nit: I ran into error messages as this:

 

As you can see (on opening the screenshot), there is missing header error from ‘/var/www/html/owncloud/apps/files_odfviewer/appinfo/app.php’. This post here told me that it could be due to a empty line. So that;s it. I went to the file and removed the line no. 8 which was an empty line! And it works fine.

GSoC 2012: On-Demand Fedora Build Service: Update #3

I have pushed a working snapshot of the image building code to github[1]. Here is a sample run of the code:

$ sudo python imagebuild.py -t boot -a i386 -o image_op1 -w image_work -p fedora -r 17 -v 1 -nvr 'anaconda-17.26-1.fc17' -bid '318281' '311809'


This command line spawns the build process of a Fedora 17 boot.iso with a number of extra packages (specified via their NVR or build IDs):

$ sudo python imagebuild.py -t boot -a i386 -o image_op1 -w image_work -p fedora -r 17 -v 1 -nvr 'anaconda-17.26-1.fc17' -bid '318281' '311809'

Downloading Extra Packages

Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
Sqlite DBs complete
Building Boot ISO
checking for root privileges
checking yum base object
setting up build architecture
setting up build parameters
installing runtime packages
running runtime-install.tmpl

downloading packages
( 1/592) [100%] downloading GConf2-3.2.5-1.fc17.i686.rpm
( 2/592) [100%] downloading ModemManager-0.5.2.0-1.fc17.i686.rpm
( 3/592) [100%] downloading NetworkManager-0.9.4.0-7.git20120403.fc17.i686.rpm
( 4/592) [100%] downloading NetworkManager-glib-0.9.4.0-7.git20120403.fc17....
( 4/592) [100%] downloading NetworkManager-glib-0.9.4.0-7.git20120403.fc17....
.
.


Finally, a boot.iso is created. Next, I plan to inegrate package retrieval from Koji via other methods. And then, support for creating Live images and DVD images. And then, the REST API/Web based service.

Some of the scripts I was experimenting with to pull packages are here [2].

[1] https://github.com/amitsaha/gsoc2012_fbs/tree/master/image_builder
[2] https://github.com/amitsaha/gsoc2012_fbs/tree/master/scripts

GSoC 2012: On-Demand Fedora Build Service: Update #1

A key component of the project is downloading packages from Koji. Over the past few days, I have been playing around with Koji client functionalities to get some familiarity with listing/retrieving packages from the build service. (Setup instructions)

Once I setup Koji, I started playing around with the client code that Tim Flink had sent me earlier. I adopted Tim’s code to create a script to download RPM’s from Koji and create a side repository with them. The Python code is called: dl_package.py.

Next, I wanted to have a script which would download the latest build of a package for a particular tag from Koji. For this, I used koji_utils.py from Autoqa‘s code. The code is called: dl_latest_build.py. As of now, this script just downloads the RPMs for each of the tags.

Both these scripts are available here: https://github.com/amitsaha/gsoc2012_fbs/tree/master/scripts

Exploring Arduino: Arduino + Ethernet + DHCP with EtherTen

In the last post on Arduino, I briefly wrote about my experiments with setting up a wireless communication between a host computer and an Arduino board using XBee. Here, we explore talking to the world of the Internet(s!) using Arduino. Time for giving your Arduino an IP address! Instead of using an Ethernet shield on an Arduino UNO, I thought the EtherTen might be a better idea to try, since it has onboard Ethernet. I am glad, I chose Etherten for various reasons!

EtherTen

Arduino Webserver

Hook up your Arduino to your computer and plug in the Ethernet cable into the board, with the other end connected to a router. If you see the Webserver example (under File -> Examples -> Ethernet) shipped with the Arduino IDE, you will see that it uses the Arduino Ethernet library.

It however requires you to set up a static IP address for your Arduino. However, in a multi-user home/university network, there is a fair chance that the IP addresses are allocated dynamically, hence I would like to use DHCP in my Webserver sketch. Now, that said: using dynamic IP address for a web server isn’t the greatest of ideas technically. But, we shall not go there in this exercise.

I used Georg Kaindl’s Arduino DHCP library to obtain the IP address dynamically and integrated with the Web server example. Here is the complete sketch. Once, you compile and load the sketch to your board, and open the Serial Monitor, you should see something like this:


Attempting to obtain a DHCP lease...
A DHCP lease has been obtained.
My IP address is 10.0.0.97
Gateway IP address is 10.0.0.138
DNS IP address is 10.0.0.138


Now, go to your web browser, and type in: 10.0.0.97:8080 and you should see:

Arduino sends the sensor readings from the Analog pins 0-5 (I don't have any sensor connected to this pins, so they are just random readings)

Now, let us use the telnet  program to send a request to our Arduino:

Telnet session

Resources and misc.

• Webserver DHCP sketch
• Tweet from your Arduino board: This sketch builds upon the example here by obtaining the IP address using DHCP
• Arduino Ethernet
• DHCP with Arduino
• Moving forward with Arduino: Ethernet
• O’Reilly’s Arduino Cookbook has some Ethernet + Networking recipes in Chapter 15.

That’s it for this time. (My previous posts on Arduino.)