GSoC 2012: The Experience

The evaluations are in and I have formally passed the Google Summer of Code, 2012 as part of the Fedora project. It was a huge learning experience. I was introduced to a number of new tools and techniques which I hadn’t heard of before.  I do not yet know how to literally “handcraft” Fedora ISO’s, however I at least know the tools for the job and have got familiar with working with Koji, the various repositories and the learnt about the “lifecycle” of a package.  In the process I hope my project will be useful to the Fedora QA, which is where we will be working towards.

• Google Melange Project page
• Project documentation

Takeaways

1. This was my first major open source project with a deadline to complete. And since I used a number of third party Open Source software, when I was stuck with a problem, the only avenue for help I had was the mailing list/project lead. That had its own shortcomings. I think there is an important lesson to learn. If you are using Open Source software in Business, you better buy support. In the same vein, if you are a Open Source software expecting to be used in anything serious, provide commercial support.
2. I had multiple priorities to deal with during the past few months – professional and personal and I wanted to do them all equally well, because all were dear to me. I am glad I managed everything pretty well in the end – Just decide what you want to do, and work towards it. Things have a strange way of coming together at the end and leave you smiling.

Thanks

Any successful endeavour is usually not achieved alone. First and foremost, thanks to Tim Flink – my project mentor for seeing this through. Thankfully, my proposed idea of a web-based build service resonated with something which he had in mind and had already worked on for a bit and that was the beginning of it all. He also had to write extra long emails explaining to me what exactly we would be attempting to achieve. Thanks, Tim for your co-operation and guidance.

Thanks are due to the community of the following projects: celery (Thanks Ask Solem and others),  Flask (Thanks guys for making me believe even I can write web applications), Fabric, Zdaemon, Read-the-docs, Sphinx and of course the Fedora community in various ways. Python made working on the project a delight. Finally, thanks Google for organizing the Summer of Code. It definitely helps in pooling a dedicated task force working on Open Source projects.

And all the non-technical factors that makes a person take up such endeavours in the first place – when that little push is needed there is always someone behind the scene giving that push- explicitly or implicitly. Thanks to Protyusha for that.

And last but not the least, sometimes you need people against you to always spur you on. So, thanks to those people who knows only how to try to pull someone down.

For those of you kids who are thinking – hey this guy is writing like he has scaled Everest, I would like to tell you: Learn to celebrate the smallest of things, it just makes it that more precious.

Advertisement

GSoC 2012: On-Demand Fedora Build Service: Final Update

The Google Summer of Code’12 working deadline was yesterday. So that brings me to this last project update.  In a last addition, I added Fedora Account System (FAS) authentication to the Web application. This was really easy, thanks to the Flask-FAS plugin. The latest documentation can be referred to here and gives you a fair idea of how you could setup and use this service.

However, this is just the end of the Google Summer of Code. The original target of this project was to make it useful to Fedora QA and that’s what we will be working towards next. So, feel free to read the docs and try it out and file issues.

GSoC 2012: On-Demand Fedora Build Service: Update #11

Its been a while since I posted an update, and this will probably be the last before the final update as part of the Google Summer of Code.

The big update is that I have got pretty decent documentation up now. You may browse it here [1]. Its up to date with the current code base [2]. Besides the user documentation, it also contains some of the internal details about the tools/libraries/design decisions that have been taken in the project.

If you have got some free cycles to spare (human + computer), you may want to give it a shot to try building  images for yourself. As a tip, the local mode, involves least amount of setup and is easiest to try out what is in there. If you see anything broken, feel free to file an issue.

As an aside, I was literally blown away by the awesomeness of Sphinx and Read the docs. Really great tools and services.

Links:

[1] Documentation: http://on-demand-fedora-build-service.readthedocs.org/en/latest/

[2] GitHub: https://github.com/amitsaha/gsoc2012_fbs

GSoC 2012: On-Demand Fedora Build Service: Update #10

In my latest commit [1], I added a number of features to the code which I had in mind:

1. Application wide Logging
2. Basic Build Monitoring
3. Email notification

These features are most useful when the service is deployed with multiple worker nodes. For local image building, these are not so important. I also added support for specifying local filesystem as staging – useful for local image building.

Next up, I have a few things to work on. Test the Web UI and Rest API interface again. Update the HOWTO (its outdated already!). And few other things remain.

Demo of Email Notification

When you submit a build request (in multiple worker nodes mode), you get an email upon notification:

Your Image Building Request have been submitted. You may monitor the progress by going to http://192.168.1.4:5100/log/tmp/imagebuild_134292350407.log. You will also recieve an email upon completion.

And then when the job completes, another email:

The build was completed by worker:: 192.168.1.4. Detailed log:

2012-07-22 12:18:24,074 – Registered a new Image Build request fromamitsaha.in@gmail.com
2012-07-22 12:18:24,075 – Image type:: dvd
2012-07-22 12:18:29,074 – Image Build notification sent
2012-07-22 12:18:29,383 – Starting the Image Build Process
2012-07-22 12:18:29,385 – DVD image arch requested should be the same as the build arch
2012-07-22 12:18:29,386 – Image building process complete
2012-07-22 12:18:29,387 – Error creating image. Transferring Logs.
2012-07-22 12:18:29,388 – Initiating local transfer of logs to /tmp/staging
2012-07-22 12:18:29,401 – Logs available at file:///tmp/staging

At this point of time, the best place to understand what is happening is to begin with the command line clients in cli/ : build_cli_basic.py and build_cli.py.I Should have an updated HOWTO soon.

Links:

[1] Latest commit: https://github.com/amitsaha/gsoc2012_fbs/commit/cf23dcc17d38924296eb099f6d1c2a440efb3d82

GSoC 2012: On-Demand Fedora Build Service: Update #9

The code saw a few additions/enhancements since the last update:

• There is now a command line client directly submitting build jobs and another client using the basic REST interface exposed via the Web application [1].
• Remote Kickstart files specified via a http:// or ftp:// are now supported. Once again, this has to be ‘ksflattened’.
• I added some validation on the web based interface. However it happens on the server side using Flask’s pre_validate( ) method. Ideally I would like to familiarize myself with enough Java Script to do this on the client side
• I started writing some Unit tests using py.test and Mock. But, I am currently waiting for some guidance from Tim to really write some good tests.

With the Mid-term evaluation due in less than a week, I am quite happy with the way things are progressing at this moment.  The TODO and GOOD-TO-HAVE notes down the things I would like to finish before the project deadline is over [2].

Links

1. https://github.com/amitsaha/gsoc2012_fbs/tree/master/cli
2. https://github.com/amitsaha/gsoc2012_fbs

GSoC 2012: On-Demand Fedora Build Service: Update #8

I spent some time cleaning up the code, mainly using pylint as an indicator. Also rewrote the fabfile to use fabric commands rather than native Linux commands wherever possible and is cleaner now. The HOWTO is also updated now.

This code refactoring has brought home (quite strongly!) the need for having my unit tests up and running soon. I plan to use py.test for my testing, and hopefully will have some tests ready in a week.

The code is available at https://github.com/amitsaha/gsoc2012_fbs

GSoC 2012: On-Demand Fedora Build Service: Update #7

In my last update, I reported that I had a functional build service, capable of harnessing multiple build nodes and a functional Web UI as well. The process to deploy the build service was however quite manual and tedious. I was looking for a way to help do this without as much manual intervention as possible and fabric (Thanks, Tim) was the answer. I wrote a fabfile for the whole task, called ‘deploy.py’.  As outlined in my earlier post, I am using celery to distribute the build tasks, so the celeryd process needs to be running on the build daemons. I found zdaemon (Thanks Jan on fabric mailing list) to be the easiest way to run the celeryd process as daemons.  The updated code is available in the repository now [1].

There is a HOWTO [2] document, which should help you deploy the service and run your own home based build service. I recently used it to build myself a Fedora Scientific ISO. Please don’t abuse it. There is hardly any error handling now. I am however accepting bug reports now. Thanks for all your help.

[1] https://github.com/amitsaha/gsoc2012_fbs/
[2] https://github.com/amitsaha/gsoc2012_fbs/blob/master/HOWTO

GSoC 2012: On-Demand Fedora Build Service: Update #6

The  code saw a number of cleanups and feature additions since the last change.

Major Additions:

• Basic Web UI functional: The web interface is quite clunky and ugly but functional. It is a Flask web application
  
• Celery for task delegation: No more SSH/SCP. The build service now uses celery (with AMQP as broker) for task delegation with workers designated as build nodes. The celery daemon however needs to run as root, since the image building process requires root acces. Configuration is specified via webapp/nodes.conf file
• FTP Image Hosting: You can specify a FTP server (enabled with anonymous login) to send your images to. The images are automatically copied to the FTP server by the code.

Build Service Web UI

Cleanups

• Single configuration file to specify the type of image to create
• No hard-coded directory structure to be created
• Miscellaneous others

High-level Functioning

1. User specifies the image creation information via the Web UI
2. This information is used to create the imagebuild.conf configuration file
3. A new process is created to delegate the task to a celery worker
4. (The celeryd needs to be running on the workers)
5. The worker process takes care of the image building and transfers the image to the specified FTP host

Todo

Here are the things I intend to work on next:

• Script to install dependencies
• Finish the cli client in webapp/
• x86_64 images
• Automatically copy the worker_src to celery workers
• Unit testing
• Implement error handling on the client UI
• Logging
• Email notification
• Enhanced UI (dynamic forms?)
• Need to think more on how the images will be stored and ability to identify an existing image uniquely. Timestamp?

Whereas, most other tasks of this project will be solved with time and effort, I am seriously concerned about the final look and feel of the Web UI. I would like it to look pretty, besides being functional.

GSoC 2012: On-Demand Fedora Build Service: Update #5

The current github code [1] does quite a few things as of now. Let me try to explain to the changes since last update and my rationale behind them.

Support for building Live images: I attempted to use livemedia-creator (which is going to be THE tool from F18+), but unfortunately ran into issues which prevented me from building images. So for now, I have implemented this feature using ‘livecd-creator’. The Kickstart file (flattened) needs to be specified and other details such as architecture, any extra packages to be pulled from Koji, etc. The specifications are specified via the config/live.conf file.

The User specifications are now completely via .conf files: Myrationale behind that in the first place was that since this code is really going to serve as the ‘backend’, command line arguments could be done away with. But, even if we want to use this as standalone, specifying .conf files is fine as well. (We will see what happens with this after discussing with my mentors). Here is a brief description of the config files in config/

• imagebuild.conf: type of image, architecture, staging area (to be explained later) and email (for notification)
• boot.conf: configuration for boot.iso images
• repoinfo.conf: repository configuration required for the above
• pungi.conf: configuration for DVD images
• live.conf: configuration for live images

The kickstart files if needed are to be placed in the kickstarts/ sub-directory. To use it, you will need to ‘cd’ into ‘image_builder’ directory and run ‘$python run_imagebuild.py’ after setting up the appropriate config files in image_builder/config and kickstart files in kickstarts/
if any.

Support for copying images: I have also now enabled support for copying the images to a ‘staging area’ as mentioned earlier. I assume a passwordless login setup and hence do a ‘scp’ once the desired image has been created. This is how you would use it standalone.

Now, as a first step towards being able to distribute build jobs to different node, I have also now added simple support for carrying out the build process on a different host. This is done by the file
delegate.py.

Here is what it does:

• Assumes that the config/ and kickstart/ files have been correctly setup by the web-form handler or manually.
• Then copies these files to the image_builder/ directory
• It creates a .tar archive of the image_builder
• Then it reads the appropriate node (architecture) from the nodes.conf file and also retrieves the working location specified.
• The .tar file is then ‘scp’-ed to the appropriate location specified
• Then runs the run_imagebuild.py script on the build node by ‘SSHing’
• The image is then automatically transferred to the staging area
  specified as earlier.

To try this feature, simply setup nodes.conf file correctly and the config/ and kickstarts/ in data/ and run $sudo python delegate.py

Note that the specified nodes should have all the dependencies installed, such as lorax, koji (setup correctly), pykickstart and livecd-creator.

[1] https://github.com/amitsaha/gsoc2012_fbs

GSoC 2012: On-Demand Fedora Build Service: Update #4

In my last update, I reported that I had the basic code to create a boot.iso including extra packages specified by the NVR or Koji build IDs.

Couple of days back, I added the support for creating a DVD iso (using ‘pungi’). Basically the code requires a pungi.conf file to be specified. For example, here is a sample pungi.conf:


[DEFAULT]
name=Fedora
ver=17
arch=i686
flavor=Desktop
destdir=/tmp/pungi_op
cachedir=/var/cache/pungi
bugurl=http://bugzilla.redhat.com
nosource=1
sourceisos=0
force=1
stage=all
config=fedora-install-fedora.ks

#Specify a working directory
workdir=/tmp/pungi_work
# specify packages via NVR
nvr=
# specify packages via Build IDs (separated by a semicolon)
bid=314456


If you are familiar with pungi, you will notice the resemblance of the configuration options with the pungi command line options. The code basically reads this configuration file and fires pungi appropriately.

As you can see, the last two options allow you to specify more recent builds of packages (via NVR and Build IDs) which are not yet available in any of the release repositories and include them instead of the less recent ones. Since pungi requires a kickstart file to be specified, I update the kickstart file by adding the side repository URL to the list of repositories specified (script here at [1]). (I have hit a problem with this step: https://www.redhat.com/archives/kickstart-list/2012-May/msg00002.html)

The updated code is now in the git repository [2] and a sample command line to build a DVD iso would be: python imagebuild.py -t dvd -a i686 (after you have created the appropriate pungi.conf file). You can also use the run_imagebuild shell script for the same.

[1] https://github.com/amitsaha/gsoc2012_fbs/blob/master/scripts/ks_addrepo.py
[2] https://github.com/amitsaha/gsoc2012_fbs/tree/master/image_builder

Thoughts:

I am beginning to think of modifying the earlier boot.iso code to read in a configuration file instead of command line arguments, like I have done for the DVD iso. This is keeping in mind the fact this code will really be the backend of the web-based build service. Hence, .conf files created at the web based frontend and sent to this image builder code would be a good way to go about it, me thinks. No? We will see.

Next up will be support for creating Live media.