Doing Math with Python: Chapters 5 and 6 in early access

I am excited to share that the fifth and sixth chapters are available as part of the early access of my book Doing Math with Python.

Chapter 5: Sets and Probability

This chapter starts off with how to create a set and demonstrating the common set operations. Utility of the different set operations are demonstrated via simple applications. For example, Cartesian product is used to write a program to simulate an experiment to calculate the time period of a simple pendulum of different lengths and at places with varying gravity. Union and intersection operations are applied to finding the probability of events.

The chapter then moves onto discussing how to generate uniform and non uniform random numbers, and using them to simulate scenarios such as a die roll and a fictional ATM which dispenses dollar bills of different denominations with varying probability.

One of the challenges at the end discusses drawing venn diagrams.

Chapter 6: Drawing shapes and Fractals

This chapter is logically divided into two parts. The first part introduces the reader to matplotlib patches which allows drawing geometric shapes (circles and polygons), followed by matplotlib’s animation API which allows drawing animated figures. The trajectory of a projectile motion discussed elsewhere in various contexts is animated combining both these things.

The second part of the book introduces the concept of geometric transformation. Combining that with the knowledge of generating random numbers learned earlier in Chapter 5, the reader will learn how to draw fractals such as the Barnsley Fern.

The challenges at the end gives the opportunity for the reader to explore the Sierpinski triangle and Henon’s function.

Trying out the programs

Using the Anaconda distribution (Python 3) should be the easiest way to try out all the programs in the book. You will need matplotlib, sympy 0.7.6 and matplotlib_venn to try out the programs. An installation guide will be available online soon.

Stay updated

I am working on the last chapter for the book. You can stay updated on the book via various channels:

Blog posts: http://echorand.me/category/doingmathwithpython/

Facebook page: https://www.facebook.com/doingmathwithpython

G+ Community: https://plus.google.com/u/0/communities/113121562865298236232

Twitter: https://twitter.com/mathwithpython

If you are interested in taking a look at a sample copy, I can try to get a sample for you to look at the current pre-released version of the book. Please feel free to get in touch.

Advertisement

Doing Math with Python: Stay Updated

I am reaching the final stages of my new book. Here are few ways to stay updated about the book:

Blog posts: http://echorand.me/category/doingmathwithpython/

Facebook page: https://www.facebook.com/doingmathwithpython

G+ Community: https://plus.google.com/u/0/communities/113121562865298236232

Twitter: https://twitter.com/mathwithpython

If you are an educator/teacher, I can also try to get a sample for you to look at the current pre-released version of the book.

Fedora 22 Scientific Alpha

Just tested the Fedora 22 Scientific Alpha RC3 image today with the test scripts/programs. Some screen shots follow:

IPython notebook

IPython notebook/SymPy/matplotlib plotting

Pandas

A complete list of all the software included is in the guide.

Contribute to Fedora Scientific

• Use it!
• You can  help complete the guide. One notable piece of software missing from that list is “pandas”.
• You can add examples/scripts/IPython notebooks to the repository here

Doing Math with Python: Two more chapters in Early Access

I am excited to share that the third and fourth chapters are available as part of the early access of my book Doing Math with Python.

Chapter 3: Describing Data with Statistics

As the title suggests, this chapter is all about the statistical measures one would first learn in high school – mean, median, mode, frequency table, range, variance, standard deviation and linear correlation are discussed.

Chapter 4: Algebra and Symbolic Math with SymPy

The first three chapters are all about number crunching. The fourth chapter introduces the reader to the basics of manipulating symbolic expressions using SymPy. Factorizing algebraic expressions, solving equations, plotting from symbolic expressions are some of the topics discussed in this chapter.

Trying out the programs

Using the Anaconda distribution (Python 3) should be the easiest way to try out all the programs in the book.

LCA 2015 talk: Beaker’s Hardware Inventory system

The video is up on YouTube: http://t.co/WorOwbv37w

Slides: https://amitksaha.fedorapeople.org/lca2015/slides.html

Since I could not make it to LCA, Nick Coghlan presented the talk on my behalf. Thanks Nick!

Announcing early access for “Doing Math with Python”

Early Access promotion:

No Starch Press is running a promotion tomorrow (29/01/2015 – PST)  that will offer 40% off all Early Access titles on nostarch.com, including Doing Math with Python. The discount code is BRIGHTANDEARLY. Currently two chapters are available and more should be up in the coming days!

I am excited to write that my new book “Doing Math with Python” to be published by No Starch Press is now available via their “Early Access” program – which means if you now buy the book, you will get the chapters as and when they are available and also, the chapters may need more polishing.

The book uses Python 3 exclusively and the Appendix A covers setup and installation instructions for Python 3 and the libraries used in the book. However, that is not yet available. Hence, at this stage, the easiest way would be to use a distribution such as Anaconda on Windows, Linux or Mac OSX (untested, I don’t have access to the OS).

C-x C-p for import pdb; pdb.set_trace()

I worked through most of the Learn Emacs Lisp in 15 minute tutorial and learned enough to write something useful:

;; Inset import pdb; pdb.set_trace() on C-x, C-p
(defun pdb-set-trace ()
  ;; http://www.emacswiki.org/emacs/InteractiveFunction
  (interactive)
  (insert "import pdb; pdb.set_trace()\n"))
(global-set-key [(control ?x) (control ?p)] 'pdb-set-trace)

Once you place this function in your Emacs init file, when you press the keys C-x and C-p, import pdb; pdb.set_trace() will magically appear in your Python program (well, anywhere for that matter, since there is no check in place). This is something which I know will be very useful to me. If you don’t like the key combination, you can change it in the last line of the above function.

If you don’t know Emacs lisp, you should try to work through the tutorial. Although, I must say I did make various attempts long time back to learn Common Lisp and Clojure, so it was not so unfamiliar to me.

Writing git hooks using Python

Since git hooks can be any executable script with an appropriate #! line, Python is more than suitable for writing your git hooks. Simply stated, git hooks are scripts which are called at different points of time in the life cycle of working with your git repository.

Let’s start by creating a new git repository:

~/work> git init git-hooks-exp
Initialized empty Git repository in /home/gene/work/git-hooks-exp/.git/
~/work> cd git-hooks-exp/
~/work/git-hooks-exp (master)> tree -al .git/
.git/
├── branches
├── config
├── description
├── HEAD
├── hooks
│   ├── applypatch-msg.sample
│   ├── commit-msg.sample
│   ├── post-update.sample
│   ├── pre-applypatch.sample
│   ├── pre-commit.sample
│   ├── prepare-commit-msg.sample
│   ├── pre-rebase.sample
│   └── update.sample
├── info
│   └── exclude
├── objects
│   ├── info
│   └── pack
└── refs
    ├── heads
    └── tags

9 directories, 12 files

Inside the .git are a number of directories and files, one of them being hooks/ which is where the hooks live. By default, you will have a number of hooks with the file names ending in .sample. They may be useful as starting points for your own scripts. However, since they all have an extension .sample, none of the hooks are actually activated. For a hook to be activated, it must have the right file name and it should be executable. Let’s see how we can write a hook using Python.

We will write a post-commit hook. This hook is called immediately after you have made a commit. We are going to do something fairly useless, but quite interesting in this hook. We will take the commit SHA1 of this commit, and print how it may look like in a more human form. I do the latter using the humanhash module. You will need to have it installed.

Here is how the hook looks like:

#!/usr/bin/python

import subprocess
import humanhash

# get the last commit SHA and print it after humanizing it
# https://github.com/zacharyvoase/humanhash
print humanhash.humanize(
    subprocess.check_output(
        ['git','rev-parse','HEAD']))

I use the subprocess.check_output() function to execute the command git rev-parse HEAD so that I can get the commit SHA1 and then call the humanhash.humanize() function with it.

Save the hook as a file, post-commit in your hooks/ directory and make it executable using chmod +x .git/hooks/post-commit. Let’s see the hook in action:

~/work/git-hooks-exp (master)> touch file
~/work/git-hooks-exp (master)> git add file
~/work/git-hooks-exp (master)> git commit -m "Added a file"
carbon-network-connecticut-equal
[master (root-commit) 2d7880b] Added a file
 1 file changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 file

The commit SHA1 for the commit turned out to be 2d7880be746a1c1e75844fc1aa161e2b8d955427. Let’s check it with the humanize function and check if we get the same message as above:

>>> humanhash.humanize('2d7880be746a1c1e75844fc1aa161e2b8d955427')
'carbon-network-connecticut-equal'

And you can see the same message above as well.

Accessing hook parameters

For some of the hooks, you will see that they are called with some parameters. In Python you can access them using the sys.argv attribute from the sys module, with the first member being the name of the hook of course and the others will be the parameters that the hook is called with.

Current working directory

For some reason, it may be useful if you know what is the current working directory of your hook. The os.getcwd() function can help there and it turns out to be the local file system path to your
git repository (~/work/git-hooks-exp in the above case).

Managing IPython notebook server via systemd: Part-I

If you are using IPython notebook on a Linux distribution which uses systemd as it’s process manager (such as Fedora Linux, Arch Linux) , you may find this post useful. I will describe a fairly basic configuration to manage (start/stop/restart) IPython notebook server using systemd.

Creating the Systemd unit file

First, we will create the systemd unit file. As root user, create a new file /usr/lib/systemd/system/ipython-notebook.service and copy the following contents into it:

[Unit]
Description=IPython notebook

[Service]
Type=simple
PIDFile=/var/run/ipython-notebook.pid
ExecStart=/usr/bin/ipython notebook --no-browser --pylab=inline
User=ipynb
Group=ipynb
WorkingDirectory=/home/ipynb/notebooks

[Install]
WantedBy=multi-user.target

Note that due to the naming of our unit file, the service will run as ipython-notebook. To completely understand the above unit file, you will need to read up a little of the topic. You may find my earlier post useful which also has links to systemd resources. Three things deserve explanation though:

The line, ExecStart=/usr/bin/ipython notebook --no-browser --pylab=inline specifies the command to start the IPython notebook server. This should be familiar to someone who uses it.

The lines, User=ipynb and Group=ipynb specify that we are going to run this process as user/group ipynb (we create them in the next step).

The line WorkingDirectory=/home/ipynb/notebooks specify that the notebooks will be stored/server in/from /home/ipynb/notebooks

Setting up the user

As root, create the user ipynb:

# useradd ipynb


Next, as ipynb, create a sub-directory, notebooks:


# su - ipynb
[ipynb@localhost ~]$ mkdir notebooks
[ipynb@localhost ~]$ exit


Starting IPython notebook

We are all set now to start IPython notebook. As the root user, reload all the systemd unit files, enable the ipython-notebook service so that it starts on boot, and then start the service:


# systemctl daemon-reload
# systemctl enable ipython-notebook
ln -s '/usr/lib/systemd/system/ipython-notebook.service' '/etc/systemd/system/multi-user.target.wants/ipython-notebook.service'
# systemctl start ipython-notebook


If you check the status of the service, it should show the following:

# systemctl status ipython-notebook
ipython-notebook.service - IPython notebook
Loaded: loaded (/usr/lib/systemd/system/ipython-notebook.service; enabled)
Active: active (running) since Sun 2013-09-22 22:39:59 EST; 23min ago
Main PID: 3671 (ipython)
CGroup: name=systemd:/system/ipython-notebook.service
├─3671 /usr/bin/python /usr/bin/ipython notebook --no-browser --pylab=inline
└─3695 /usr/bin/python -c from IPython.zmq.ipkernel import main; main() -f /home/ipynb/.ipython/profile_default/security/kernel-6dd8b338-e779-4e67-bf25-1cd238...

Sep 22 22:39:59 localhost ipython[3671]: [NotebookApp] Serving notebooks from /home/ipynb/notebooks
Sep 22 22:39:59 localhost ipython[3671]: [NotebookApp] The IPython Notebook is running at: http://127.0.0.1:8888/
Sep 22 22:39:59 localhost ipython[3671]: [NotebookApp] Use Control-C to stop this server and shut down all kernels.
Sep 22 22:40:21 localhost ipython[3671]: [NotebookApp] Using MathJax from CDN: http://cdn.mathjax.org/mathjax/latest/MathJax.js
Sep 22 22:40:22 localhost ipython[3671]: [NotebookApp] Kernel started: 6dd8b338-e779-4e67-bf25-1cd23884cf5a
Sep 22 22:40:22 localhost ipython[3671]: [NotebookApp] Connecting to: tcp://127.0.0.1:51666
Sep 22 22:40:22 localhost ipython[3671]: [NotebookApp] Connecting to: tcp://127.0.0.1:52244
Sep 22 22:40:22 localhost ipython[3671]: [NotebookApp] Connecting to: tcp://127.0.0.1:44667
Sep 22 22:40:22 localhost ipython[3671]: [IPKernelApp] To connect another client to this kernel, use:
Sep 22 22:40:22 localhost ipython[3671]: [IPKernelApp] --existing kernel-6dd8b338-e779-4e67-bf25-1cd23884cf5a.json

You should now be able to access IPython notebook as you would normally do. Finally, you can stop the server as follows:


# systemctl stop ipython-notebook


The logs are redirected to /var/log/messages:


Sep 22 22:39:59 localhost ipython[3671]: [NotebookApp] Created profile dir: u'/home/ipynb/.ipython/profile_default'
Sep 22 22:39:59 localhost ipython[3671]: [NotebookApp] Serving notebooks from /home/ipynb/notebooks
Sep 22 22:39:59 localhost ipython[3671]: [NotebookApp] The IPython Notebook is running at: http://127.0.0.1:8888/
Sep 22 22:39:59 localhost ipython[3671]: [NotebookApp] Use Control-C to stop this server and shut down all kernels.
Sep 22 22:40:21 localhost ipython[3671]: [NotebookApp] Using MathJax from CDN: http://cdn.mathjax.org/mathjax/latest/MathJax.js
Sep 22 22:40:22 localhost ipython[3671]: [NotebookApp] Kernel started: 6dd8b338-e779-4e67-bf25-1cd23884cf5a
Sep 22 22:40:22 localhost ipython[3671]: [NotebookApp] Connecting to: tcp://127.0.0.1:51666
Sep 22 22:40:22 localhost ipython[3671]: [NotebookApp] Connecting to: tcp://127.0.0.1:52244
Sep 22 22:40:22 localhost ipython[3671]: [NotebookApp] Connecting to: tcp://127.0.0.1:44667
Sep 22 23:05:35 localhost ipython[3671]: [NotebookApp] received signal 15, stopping
Sep 22 23:05:35 localhost ipython[3671]: [NotebookApp] Shutting down kernels
Sep 22 23:05:35 localhost ipython[3671]: [NotebookApp] Kernel shutdown: 6dd8b338-e779-4e67-bf25-1cd23884cf5a

For me, the biggest reason to do this is that I do not have to start the IPython notebook server everytime on system startup manually, since I know it will be running when I need to use it. I plan to explore managing custom profiles next and also think more about a few other things.

Notes on writing systemd unit files for Beaker’s daemon processes

Recently, I had a chance to write systemd unit files for the daemon processes that run as part of Beaker: beakerd which is the scheduling daemon running on the server and the four daemons running on the lab controller – beaker-proxy, beaker-provision, beaker-watchdog and beaker-transfer.

This post may be of interest to you if you are using python-daemon to write programs which are capable of running as daemon processes and you want to write systemd unit files for them.

beakerd’s unit file

Here is the systemd unit file for beakerd, which I will use to illustrate the core points of this post. The other unit files are similar, and hence I will explain only where they differ from this one:

[Unit]
Description=Beaker scheduler
After=mysqld.service

[Service]
Type=forking
PIDFile=/var/run/beaker/beakerd.pid
ExecStart=/usr/bin/beakerd
User=apache
Group=apache

[Install]
WantedBy=multi-user.target

The [Unit] section has a description of the service (using the Description option) and specifies that it should start after the mysqld.service has started using the After option. beakerd needs to communicate to a MySQL server before it can start successfully. It can work with a local or a remote MySQL server. Hence, specifying After sets up an ordering that if there is a local MySQL server, then wait for it to start before starting beakerd. Using Requires is not suitable here to accommodate the possibility that beakerd may be configured to use a remote MySQL server.

In the [Service] section, the Type is set to Forking. This is because, beakerd uses python-daemon which forks itself (detaches itself) during the daemonization. However, you must ensure that when creating a DaemonContext() object, you should specify detach_process=True. This is because, if python-daemon detects that it is running under a init manager, it doesn’t detach itself unless the keyword is explicitly set to True, as above (you can see the code in daemon.py). Hence, although not setting the above keyword would work under SysV Init, it doesn’t work under systemd (with Type=Forking), since the daemon doesn’t fork at all and systemd expects it to fork (and finally kills it). The PIDFile specifies where the process ID is dumped by beakerd and is setup while creating the DaemonContext object as follows and ExecStart specifies the location to the binary that is to be started.

The beakerd process is to be run as the apache user and group, which is specified by the User and Group options.

In the [Install] section, the WantedBy option specifies when the beakerd process should be started (similar to the concept of “run levels” in SysV init). systemd defines several targets, and here we define that we want beakerd to start as part of the multi user setup.

That’s all about beakerd’s unit file.

beaker-provision’s unit file

beaker-provision and the other daemons running on the lab controller have similar unit files:

[Unit]
Description=Beaker provisioning daemon
After=httpd.service

[Service]
Type=forking
PIDFile=/var/run/beaker-lab-controller/beaker-provision.pid
ExecStart=/usr/bin/beaker-provision
User=root
Group=root

[Install]
WantedBy=multi-user.target

All the four lab controller daemons need to communicate with Beaker web application – which can be local or remote, and hence the After option specifies the dependency on httpd.service. And, this particular daemon runs as root user/group, which is specified by the User and group options.

And everything else is similar to beakerd’s unit file and also the other lab controller daemons.

Shipping SysV init files and systemd unit files in the same package

The beaker packages ship both SysV init files and systemd unit files now so that it can use systemd when available, but use SysV init otherwise. This commit can give you some idea of how to go about it.

systemd resources

These links proved helpful to learn more about systemd, including how to package unit files for Fedora:

• http://www.freedesktop.org/wiki/Software/systemd/
• http://fedoraproject.org/wiki/Systemd
• http://www.freedesktop.org/software/systemd/man/systemd.unit.html
• http://fedoraproject.org/wiki/Packaging:Systemd
• http://fedoraproject.org/wiki/Packaging:ScriptletSnippets#Systemd
• http://freedesktop.org/wiki/Software/systemd/Preset/
• http://www.freedesktop.org/wiki/Software/systemd/TipsAndTricks/
• http://www.freedesktop.org/software/systemd/man/daemon.html
• http://www.pontikis.net/blog/migration-from-initscripts-to-systemd-on-archlinux
• http://0pointer.de/blog/projects/why.html