Just tested the Fedora 22 Scientific Alpha RC3 image today with the test scripts/programs. Some screen shots follow:
IPython notebook
IPython notebook/SymPy/matplotlib plotting
Pandas
A complete list of all the software included is in the guide.
Contribute to Fedora Scientific
The video is up on YouTube: http://t.co/WorOwbv37w
Slides: https://amitksaha.fedorapeople.org/lca2015/slides.html
Since I could not make it to LCA, Nick Coghlan presented the talk on my behalf. Thanks Nick!
While working with beaker‘s code base, I often feel the need to run my tests for a patch/feature and continue to work on with different things while they run, including running other tests testing something different. Currently this is not possible since we start off with a clean database on every test run and simultaneous runs would obviously make one run step on another’s feet.
I finally have an initial docker based prototype for making this possible.
Fedora 20 is now released, which also means the newest release of Fedora Scientific along with other spins are also available.
Download
You can download the spin images from here.
What’s new in Fedora Scientific
The notable additions in this release are:
The Fedora 20 release notes are here.
Fedora Scientific Documentation
I started work on some documentation for Fedora Scientific about a month or so back. It is far from what I want it to be, but you can see the current version here. The first goal that I have in mind is to document all the major scientific tools and libraries that are shipped with Fedora Scientific. By document, I imply links to the official project resources and guides. The second goal is to actually add original content and make it a guide book for Fedora Scientific which may be used as an entry point for Open Source Scientific Computing. Once the guide has taken some shape, an RPM package can be created and distributed with Fedora Scientific so that the entire documentation is available for offline perusal.
Contributing
The Fedora Scientific documentation is an excellent starting point if you are looking to make a contribution to Fedora Scientific. You can view the project here.
If you have some toy/throwaway scripts that makes use of one of the libraries/tools, you may want to contribute it to the
“tests” here. They will help sanity check these libraries and tools during the development of upcoming Fedora Scientific releases.
Discussions and support
Please join the Fedora scitech mailing list.
Suggetions and ideas? Please leave a comment.
If you are using IPython notebook on a Linux distribution which uses systemd as it’s process manager (such as Fedora Linux, Arch Linux) , you may find this post useful. I will describe a fairly basic configuration to manage (start/stop/restart) IPython notebook server using systemd.
Creating the Systemd unit file
First, we will create the systemd unit file. As root user, create a new file /usr/lib/systemd/system/ipython-notebook.service and copy the following contents into it:
[Unit]
Description=IPython notebook
[Service]
Type=simple
PIDFile=/var/run/ipython-notebook.pid
ExecStart=/usr/bin/ipython notebook --no-browser --pylab=inline
User=ipynb
Group=ipynb
WorkingDirectory=/home/ipynb/notebooks
[Install]
WantedBy=multi-user.target
Note that due to the naming of our unit file, the service will run as ipython-notebook. To completely understand the above unit file, you will need to read up a little of the topic. You may find my earlier post useful which also has links to systemd resources. Three things deserve explanation though:
The line, ExecStart=/usr/bin/ipython notebook --no-browser --pylab=inline specifies the command to start the IPython notebook server. This should be familiar to someone who uses it.
The lines, User=ipynb and Group=ipynb specify that we are going to run this process as user/group ipynb (we create them in the next step).
The line WorkingDirectory=/home/ipynb/notebooks specify that the notebooks will be stored/server in/from /home/ipynb/notebooks
Setting up the user
As root, create the user ipynb:
# useradd ipynb
Next, as ipynb, create a sub-directory, notebooks:
# su - ipynb
[ipynb@localhost ~]$ mkdir notebooks
[ipynb@localhost ~]$ exit
Starting IPython notebook
We are all set now to start IPython notebook. As the root user, reload all the systemd unit files, enable the ipython-notebook service so that it starts on boot, and then start the service:
# systemctl daemon-reload
# systemctl enable ipython-notebook
ln -s '/usr/lib/systemd/system/ipython-notebook.service' '/etc/systemd/system/multi-user.target.wants/ipython-notebook.service'
# systemctl start ipython-notebook
If you check the status of the service, it should show the following:
# systemctl status ipython-notebook
ipython-notebook.service - IPython notebook
Loaded: loaded (/usr/lib/systemd/system/ipython-notebook.service; enabled)
Active: active (running) since Sun 2013-09-22 22:39:59 EST; 23min ago
Main PID: 3671 (ipython)
CGroup: name=systemd:/system/ipython-notebook.service
├─3671 /usr/bin/python /usr/bin/ipython notebook --no-browser --pylab=inline
└─3695 /usr/bin/python -c from IPython.zmq.ipkernel import main; main() -f /home/ipynb/.ipython/profile_default/security/kernel-6dd8b338-e779-4e67-bf25-1cd238...
Sep 22 22:39:59 localhost ipython[3671]: [NotebookApp] Serving notebooks from /home/ipynb/notebooks
Sep 22 22:39:59 localhost ipython[3671]: [NotebookApp] The IPython Notebook is running at: http://127.0.0.1:8888/
Sep 22 22:39:59 localhost ipython[3671]: [NotebookApp] Use Control-C to stop this server and shut down all kernels.
Sep 22 22:40:21 localhost ipython[3671]: [NotebookApp] Using MathJax from CDN: http://cdn.mathjax.org/mathjax/latest/MathJax.js
Sep 22 22:40:22 localhost ipython[3671]: [NotebookApp] Kernel started: 6dd8b338-e779-4e67-bf25-1cd23884cf5a
Sep 22 22:40:22 localhost ipython[3671]: [NotebookApp] Connecting to: tcp://127.0.0.1:51666
Sep 22 22:40:22 localhost ipython[3671]: [NotebookApp] Connecting to: tcp://127.0.0.1:52244
Sep 22 22:40:22 localhost ipython[3671]: [NotebookApp] Connecting to: tcp://127.0.0.1:44667
Sep 22 22:40:22 localhost ipython[3671]: [IPKernelApp] To connect another client to this kernel, use:
Sep 22 22:40:22 localhost ipython[3671]: [IPKernelApp] --existing kernel-6dd8b338-e779-4e67-bf25-1cd23884cf5a.json
You should now be able to access IPython notebook as you would normally do. Finally, you can stop the server as follows:
# systemctl stop ipython-notebook
The logs are redirected to /var/log/messages:
Sep 22 22:39:59 localhost ipython[3671]: [NotebookApp] Created profile dir: u'/home/ipynb/.ipython/profile_default'
Sep 22 22:39:59 localhost ipython[3671]: [NotebookApp] Serving notebooks from /home/ipynb/notebooks
Sep 22 22:39:59 localhost ipython[3671]: [NotebookApp] The IPython Notebook is running at: http://127.0.0.1:8888/
Sep 22 22:39:59 localhost ipython[3671]: [NotebookApp] Use Control-C to stop this server and shut down all kernels.
Sep 22 22:40:21 localhost ipython[3671]: [NotebookApp] Using MathJax from CDN: http://cdn.mathjax.org/mathjax/latest/MathJax.js
Sep 22 22:40:22 localhost ipython[3671]: [NotebookApp] Kernel started: 6dd8b338-e779-4e67-bf25-1cd23884cf5a
Sep 22 22:40:22 localhost ipython[3671]: [NotebookApp] Connecting to: tcp://127.0.0.1:51666
Sep 22 22:40:22 localhost ipython[3671]: [NotebookApp] Connecting to: tcp://127.0.0.1:52244
Sep 22 22:40:22 localhost ipython[3671]: [NotebookApp] Connecting to: tcp://127.0.0.1:44667
Sep 22 23:05:35 localhost ipython[3671]: [NotebookApp] received signal 15, stopping
Sep 22 23:05:35 localhost ipython[3671]: [NotebookApp] Shutting down kernels
Sep 22 23:05:35 localhost ipython[3671]: [NotebookApp] Kernel shutdown: 6dd8b338-e779-4e67-bf25-1cd23884cf5a
For me, the biggest reason to do this is that I do not have to start the IPython notebook server everytime on system startup manually, since I know it will be running when I need to use it. I plan to explore managing custom profiles next and also think more about a few other things.
Beaker 0.14 was released recently and if you are an existing user of Beaker, you may see the What’s new page here.
If however, you do not know what Beaker is, the Architecture guide is a good start and if things look interesting, with this release there is also documentation now to setup a Beaker “test bed” using two Virtual machines (via libvirt).
Recently, I had a chance to write systemd unit files for the daemon processes that run as part of Beaker: beakerd which is the scheduling daemon running on the server and the four daemons running on the lab controller – beaker-proxy, beaker-provision, beaker-watchdog and beaker-transfer.
This post may be of interest to you if you are using python-daemon to write programs which are capable of running as daemon processes and you want to write systemd unit files for them.
beakerd’s unit file
Here is the systemd unit file for beakerd, which I will use to illustrate the core points of this post. The other unit files are similar, and hence I will explain only where they differ from this one:
[Unit] Description=Beaker scheduler After=mysqld.service [Service] Type=forking PIDFile=/var/run/beaker/beakerd.pid ExecStart=/usr/bin/beakerd User=apache Group=apache [Install] WantedBy=multi-user.target
The [Unit] section has a description of the service (using the Description option) and specifies that it should start after the mysqld.service has started using the After option. beakerd needs to communicate to a MySQL server before it can start successfully. It can work with a local or a remote MySQL server. Hence, specifying After sets up an ordering that if there is a local MySQL server, then wait for it to start before starting beakerd. Using Requires is not suitable here to accommodate the possibility that beakerd may be configured to use a remote MySQL server.
In the [Service] section, the Type is set to Forking. This is because, beakerd uses python-daemon which forks itself (detaches itself) during the daemonization. However, you must ensure that when creating a DaemonContext() object, you should specify detach_process=True. This is because, if python-daemon detects that it is running under a init manager, it doesn’t detach itself unless the keyword is explicitly set to True, as above (you can see the code in daemon.py). Hence, although not setting the above keyword would work under SysV Init, it doesn’t work under systemd (with Type=Forking), since the daemon doesn’t fork at all and systemd expects it to fork (and finally kills it). The PIDFile specifies where the process ID is dumped by beakerd and is setup while creating the DaemonContext object as follows and ExecStart specifies the location to the binary that is to be started.
The beakerd process is to be run as the apache user and group, which is specified by the User and Group options.
In the [Install] section, the WantedBy option specifies when the beakerd process should be started (similar to the concept of “run levels” in SysV init). systemd defines several targets, and here we define that we want beakerd to start as part of the multi user setup.
That’s all about beakerd’s unit file.
beaker-provision’s unit file
beaker-provision and the other daemons running on the lab controller have similar unit files:
[Unit] Description=Beaker provisioning daemon After=httpd.service [Service] Type=forking PIDFile=/var/run/beaker-lab-controller/beaker-provision.pid ExecStart=/usr/bin/beaker-provision User=root Group=root [Install] WantedBy=multi-user.target
All the four lab controller daemons need to communicate with Beaker web application – which can be local or remote, and hence the After option specifies the dependency on httpd.service. And, this particular daemon runs as root user/group, which is specified by the User and group options.
And everything else is similar to beakerd’s unit file and also the other lab controller daemons.
Shipping SysV init files and systemd unit files in the same package
The beaker packages ship both SysV init files and systemd unit files now so that it can use systemd when available, but use SysV init otherwise. This commit can give you some idea of how to go about it.
systemd resources
These links proved helpful to learn more about systemd, including how to package unit files for Fedora:
In my last post, I explained how you could upgrade your Fedora 18 Scientific installation to Fedora 19 scientific installation. However, if you are looking to perform a fresh installation, you would need a ISO image. I will now show you how you create an ISO image yourself.
Please note that you will need a Fedora 19 installation to create the Fedora 19 scientific image. Also, you should keep in mind that the architecture of the ISO that you build is the same as the architecture of the system you build it on.
The steps are as follows:
Install the necessary tools
We will be using the program, livecd-creator to create the ISO which is installed by the livecd-tools package. Hence install this package using:
#yum install livecd-tools
Clone the kickstarts repository
The Fedora Scientific spin and other spins and images are created from a set of kickstart files maintained in the spin-kickstarts.git repository. Clone this repository (we will directly clone the ‘f19’ branch, since we will be generating a Fedora 19 image). The command for cloning is:
# git clone -b f19 git://git.fedorahosted.org/spin-kickstarts.git
Build the image
Before you can build the image, set SELinux to permissive mode:
# setenforce 0
Now, change your current working directory to the spin-kickstarts directory and invoke the livecd-creator program as follows:
# livecd-creator fedora-livedvd-scientific-kde.ks
The entire process is fairly time consuming, pulls in a lot of packages from the repositories (network data consuming) and disk space intensive. Make sure you have access to all of them. Once the build process is complete, you should have a .iso file in the same directory (The file name will be something like livecd-fedora-livedvd-scientific-kde-xxyyzz.iso). You can now proceed with installing it in a virtual machine or burning it to a USB stick and installing it on a real machine.
Resources
Fedora 19 was released on July 2, 2013. The Fedora Scientific spin is unfortunately not available for this release, since I failed to remove couple of missing packages (scite and netbeans) from the kickstart file and hence the build was failing at the cut-off time. My apologies to all those old users who were looking forward to do a fresh install of Fedora 19 Scientific and the new users who were looking to try it out.
In this post I will describe to you in a few steps how you can upgrade your existing Fedora 18 Scientific installation to Fedora 19. We will use the FedUp utility for this purpose. Since this is a upgrade, the usual warnings apply. Things may go wrong, so please take backups of your data and configuration files if they are important to you. It is a good way to read the FedUp page before proceeding to get a general idea of what’s involved.
Here are the steps (standard for any Fedora upgrade adopted from the FedUp guide):
Update your system
First and foremost, update your system (# yum update) and reboot to the latest kernel.
Install FedUp
Install the latest fedup package (# yum install fedup --enablerepo=updates-testing)
Prepare for the upgrade
If you see the FedUp guide, you will see that there are more than one way of preparing for the upgrade. I used the recommended network method. Here is what you have to do:
# fedup-cli --network 19 --instrepo http://dl.fedoraproject.org/pub/fedora/linux/releases/19/Fedora/x86_64/os/ --addrepo Everything=http://dl.fedoraproject.org/pub/fedora/linux/releases/19/Everything/x86_64/os/
Substitute the instrepo URL by something closer to you. I also found that I needed to add the Everything repo to get all the packages that Fedora Scientific has installed. If this step completes for you without any errors, reboot your system.
Executing the Upgrade
You should see a System Upgrade menu item in your GRUB menu, hit ENTER on that and your system should show the Plymouth screen. You can hit the ESC key to see what is happening. You should see the packages being upgraded. Once it completes, it should reboot and you can now boot into Fedora 19.
And you should have Fedora 19 with all the scientific spin packages updated to the latest releases.
Next steps
I didn’t need to do anything else.
I needed to do a # yum distro-sync to remove some of the F18 packages and get the F19 ones (emacs, for example).
However, I performed all of this in a virtual machine and I also did not upgrade my GRUB. (See here for the instructions).They worked for me. If it doesn’t work for you, leave a comment and I will try to address it.
New Installation of Fedora 19 Scientific
If you want to perform a fresh install of Fedora 19 scientific, it is easy to build your own live image. I will show in a next post to come up soon.
Here is the post: http://echorand.me/2013/07/04/creating-a-fedora-19-scientific-iso/
In this article, we will explore the Python programming language as a tool to retrieve various information about a system running Linux. We start by exploring the platform module. For example, the platform.uname() function returns similar information to the Linux command, uname:
>>> import platform
>>> platform.uname()
('Linux', 'fedora.echorand', '3.7.4-204.fc18.x86_64', '#1 SMP Wed Jan 23 16:44:29 UTC 2013', 'x86_64')
Read the entire article here
Programming and writing about it. 