Book Review: Linux System Programming

I received a review copy of the book as part of the blogger review program.

Title:  “Linux System Programming (2nd Edition) ” by Robert Love; O’Reilly Media.

Summary:  

This book consists of 11 chapters. The first chapter introduces you nicely to the the core topics and lays the foundation for the rest of the book. Files (including some hints on the role of the virtual file system and how they are represented in the Kernel), Input/Output (User buffered I/O, I/O scheduling, Scatter-Gather I/O), Processes (including their creation mechanisms and management), Threads (and how Linux implements them along with a treatment of the POSIX threads library), Memory (Process address space, dynamic memory allocation strategies, and how they work, memory locking) form the core of the book. The second last chapter discusses signal handling. The last chapter of the book is on time (the different types of time, how you can get/set time, measure time elapsed and timers) and is sort of a “standalone” topic for the book. The first appendix discusses the GCC extensions to the C language and can be handy when you read the Kernel source code.

Reactions:

In this book, the author discusses some of the most important topics that one would want to learn about when venturing into the area of “system programming” on Linux. He introduces the topics in a friendly manner adding some fun anecdotes from time to time (what does the “c” in calloc() stand for?).At various places, the reader is given a peek under the hood (for example, pause() is one of the simplest system calls implemented) which can only make the curious reader happy and itchy to download the kernel source code and start grepping. The book includes code examples throughout and hence if you are learning a topic for the first time, these are very useful starting points.

Verdict: 

System programming on Linux is an area encompassing number of related topics most of which can fill up whole books on their own. I also could not help comparing this book with “The Linux Programming Interface” by Michael Kerrisk (a book which I own already). Should you buy this book if you already own the latter? Yes, you should. While not being “encyclopedic” and not covering topics such as socket programming at all, Robert Love’s “Linux System Programming” has the right level of treatment and detail for the reader interested in system programming.

Product page:

• O’Reilly product page

Advertisement

poweroff, halt, reboot and systemctl

On Fedora (and perhaps other Linux distros using systemd) you will see that the poweroff, reboot and halt commands are all symlinks to systemctl:

> ls -l /sbin/poweroff /sbin/halt /sbin/reboot 
lrwxrwxrwx. 1 root root 16 Oct  1 11:04 /sbin/halt -> ../bin/systemctl
lrwxrwxrwx. 1 root root 16 Oct  1 11:04 /sbin/poweroff -> ../bin/systemctl
lrwxrwxrwx. 1 root root 16 Oct  1 11:04 /sbin/reboot -> ../bin/systemctl

So, how does it all work? The answer lies in this code block from systemctl.c:

..
5556        if (program_invocation_short_name) {                                                                                                                                              
5557                                                                                                                                                                                          
5558                if (strstr(program_invocation_short_name, "halt")) {                                                                                                                      
5559                        arg_action = ACTION_HALT;                                                                                                                                         
5560                        return halt_parse_argv(argc, argv);                                                                                                                               
5561                } else if (strstr(program_invocation_short_name, "poweroff")) {                                                                                                           
5562                        arg_action = ACTION_POWEROFF;                                                                                                                                     
5563                        return halt_parse_argv(argc, argv);                                                                                                                               
5564                } else if (strstr(program_invocation_short_name, "reboot")) {                                                                                                             
5565                        if (kexec_loaded())                                    
..

program_invocation_short_name
program_invocation_short_name is a variable (GNU extension) which contains the name used to invoke a program. The short indicates that if you call your program as /bin/myprogram, it is set to ‘myprogram’. There is also a program_invocation_name variable consisting of the entire path. Here is a demo:

/*myprogram.c*/

# include <stdio.h>

extern char *program_invocation_short_name;
extern char *program_invocation_name;

int main(int argc, char **argv)
{
  printf("%s \n", program_invocation_short_name);
  printf("%s \n", program_invocation_name);
  return 0;
}

Assume that the executable for the above program is created as myprogram, execute the program from a directory which is one level up from where it resides. For example, in my case, myprogram is in $HOME/work and I am executing it from $HOME:

> ./work/myprogram 
myprogram 
./work/myprogram 

You can see the difference between the values of the two variables. Note that any command line arguments passed are not included in any of the variables.

Back to systemctl

Okay, so now we know that when we execute the poweroff command (for example), program_invocation_short_name is set to poweroff and this check matches:

if (strstr(program_invocation_short_name, "poweroff")) 
..

and then the actual action of powering down the system takes place. Also note that how the halt_parse_argv function is called with the parameters argc and argv so that when you invoke the poweroff command with a switch such as --help, it is passed appropriately to halt_parse_argv:

5194        static const struct option options[] = {                                                                                                                                          
5195                { "help",      no_argument,       NULL, ARG_HELP    },                                                                                                                    
..
..
5218                case ARG_HELP:                                                                                                                                                            
5219                        return halt_help(); 

Fooling around

Considering that systemctl uses strstr to match the command it was invoked as, it allows for some fooling around. Create a symlink mypoweroff to /bin/systemctl and then execute it as follows:

> ln -s /bin/systemctl mypoweroff
> ./mypoweroff --help
mypoweroff [OPTIONS...]

Power off the system.

     --help      Show this help
     --halt      Halt the machine
  -p --poweroff  Switch off the machine
     --reboot    Reboot the machine
  -f --force     Force immediate halt/power-off/reboot
  -w --wtmp-only Don't halt/power-off/reboot, just write wtmp record
  -d --no-wtmp   Don't write wtmp record
     --no-wall   Don't send wall message before halt/power-off/reboot

This symlink is for all purpose going to act like the poweroff command since systemctl basically checks whether ‘poweroff’ is a substring of the invoked command.

To learn more, see systemctl.c

Related

Few months back, I had demoed invoking a similar behaviour in programs where a program behaves differently based on how you invoke it using argv[0] here. I didn’t know of the GNU extensions back then.

Using the __cleanup__ variable attribute in GCC

GCC’s C compiler allows you to define various variable attributes. One of them is the cleanup attribute (which you can also write as__cleanup__) which allows you to define a function to be called when the variable goes out of scope (for example, before returning from a function). This is useful, for example to never forget to close a file or freeing the memory you may have allocated. Next up is a demo example defining this attribute on an integer variable (which obviously has no practical value). I am using gcc (GCC) 4.7.2 20121109 on Fedora 18.

Read the article here.

New Book: Write Your First Program – C and Python

I am excited to announce that my book, Write Your First Program is finally available for public consumption. In this day and age, when there are easily accessible programming books galore (in the context of both, price and availability), why write a new book? Through this post, I hope to reach out to the prospective readers and buyers (and hence critics) of this book.

I will start with a summary of the contents of the book, and then make an attempt to analyze a few key things I chose to be a part of this book and how I think those choices should help anyone attempting to get a first taste of programming today.

Summary of the book

The book stands at 242 pages (in paperback). It has seven major chapters starting from a Hello World program, followed by the concepts of variables, input/output, user-defined functions, data structures, file handling, mathematical functions and advanced concepts (object oriented programming, error handling and version control) . Each chapter discusses and implements each of those concepts in two programming languages: C and Python.  Most of the chapters have a programming fun section where I use the concepts introduced in that chapter (and others before that) to write a program which the readers can enjoy implementing. There are exercise problems at the end of each chapter which aims to encourage the reader to try out writing new programs.

Analysis of the Choices

C & Python: Besides the contents, which I will come to soon, the major decision I made while starting with the book is that it will aim to teach not one, but two programming languages – C and Python. The choice was far from being arbitrary or a way to be different from most programming books. It reflects my own (however flawed that may turn out to be!) assessment of what the beginning programmer today should learn.

I was exposed to C few years after I learned to program and it’s power became clear to me as I explored writing MS-DOS TSRs and later, Linux Kernel modules. Somewhere in college, in the early 2000’s, I got a book on Python. As I have programmed increasingly in Python and learned of the multitude of ways it’s been used in all sorts of projects – Linux utilities, Scientific and Numerical computing and Web application programming, my quest to learn more about Python increased and with that my urge to talk and write about how awesome Python is. For me C was raw power, where as Python was elegant and beautiful – good to read, write in and get the job done fast.

And hence, when finally the moment arrived I decided that I would attempt to teach both C and Python in my first book.  I have this urge in me to always get out all I know and share with someone (not verbally, I am not much of a talker) via my writings.

How would I go about teaching two different programming languages in one book without intimidating the audience of the book?

Contents: I decided that I would include topics which covers the bare minimum that the beginning programmer should know from the perspective of programming. I start the book introducing concepts such as variables, data types, input and output, control structures and user-defined functions, showing how to implement the ideas and concepts as programs in both the languages. I introduced C’s scary pointers in Chapter 2, along with the discussion on variables and data types. It just seemed natural to me that addresses of variables should be discussed with variables. I also introduce recursion in the chapter on user defined functions. Towards the end, I cover the topics of encoding, error handling, and working with multiple files. Considering the importance of using version control for projects, I also discuss the preliminaries of the git version control system and briefly touch upon the idea of text encoding in one of the appendices. Overall, I aimed to keep the topics naturally coherent not assuming a topic or concept may be difficult for the beginning programmer and hence introduce it later. The reader should ideally skip such a topic and come back to later.

Within each topic, I tried to keep the treatment to “just enough”, but not superficial. The reader will be able to learn enough to write useful programs (as part of an academic course, hobby project) and more importantly, be able to approach more advanced areas of either programming language with confidence. I also refrained from discussing quirks of programming language constructs (such as for(;;) in C) , which I believe can be lived without. There are books out there aiming to equip a reader with such knowledge and honestly, even if I wanted to, I don’t have the skills to do that. My book doesn’t aim to be a reference text. It aims to get you started, true to its title – help you write your first program.

Throughout the book, I have attempted to discuss programming in the context of other “subjects” of study – by asking the reader to attempt writing programming solutions to problems from those fields. This will hopefully also interest the reader who may not aim to take up computer science or programming as a career, but may need to use it as a tool.

Well, that pretty much sums it up. If you think this book may be worth a read, the buying options are as follows.

Buying the book

Print copy: Print copies are limited to India only (at least as of now). You can buy it online using any of the options below:

• Amazon India
• Flipkart
• Prentice Hall Learning India’s website
• Amazon

Digital Editions: 

• Amazon Kindle (Even if you don’t have a Kindle, you can read it using Amazon’s Kindle applications. In fact, the book’s source code doesn’t look so good on the “old” Kindle, so, I recommend the kindle applications strongly)
• Adobe Digital Editions

Closing Notes

The book’s website is at http://phindia.com/saha and it will contain additional articles/materials and other content that the book’s readers may find useful. And of course, a list of errata (already found two myself) :-). Thanks for reading this post. I would be keen to hear from you in case you happen to buy my book or have something related to say/discuss.

This book was quite a journey on the personal front as well. You may enjoy reading it here.

Book Review: Understanding and Using C Pointers

I received a review copy of the book as part of the blogger review program.

Title:  “Understanding and Using C Pointers” by Richard Reese; O’Reilly Media.

Summary:  As the title implies, in the book “Understanding and Using C Pointers”, the author discusses pointers in C. It discusses various aspects of pointers and their application in dynamic memory allocation (and implementing linked lists, dynamic arrays, etc.), their usage in the form of function pointers, structure pointers and various other related concepts. The treatment of the topics have a nice flow. The author gently begins by introducing the concepts of a stack and a heap in the context of the runtime memory layout of a executing program and then proceeds to discuss memory allocation, before discussing the topics mentioned earlier. The book’s contents describe an area of C which can be scary and confusing sometimes, but need not necessarily be so – which is what the book aims to demonstrate.

Reactions: The book is absolutely fantastic and if you have anything to do with C or simply wanted to read an entire book on pointers, this is it. You will  learn a number of new things, and suddenly a number of pointer mysteries will be demystified.

Verdict: A must have for your collection if you would love to learn more about pointers in C or simply understand them better.

Product Page(s):

• O’Reilly
• Amazon

Sphinx as a Static Site Generator

In this article, we will take a look at Sphinx (specifically Sphinx 1.1.3) as a static
site generator tool. Let’s get started.

Installing Sphinx

Installing sphinx is as easy as pip install sphinx (on Linux and
Windows). pip will download a bunch of other dependencies (docutils,
Jinja2, pygments as well). Once installed, check whether sphinx is
available to you by typing $ sphinx-quickstart at the command
line. This should give you a welcome message. We proceed from there in
the next section. (On Windows, if you have added
C:\Python27\Scripts to the Windows Path variable, the sphinx
commands should be already available to be executed since the
sphinx binaries are also installed in this location).

Read the entire article here.

Compilation and Interpretation in C and CPython

It is common knowledge that programs written in high level languages
have to be translated into a low level language using programs
referred to as translators. This low level language is either in a native form, in the sense that it is understood by the operating system itself, or in an intermediate
form which is understood by an intermediate program such as the bytecode
interpreter. It is also known that C is a compiled language, whereas
CPython is both first compiled and then interpreted.

Read the entire article here.

Data in C and CPython: A comparison

In my last two articles (Data in CPython and Data in C), as it
turned out, I discussed two fundamental points in each language:

• What happens in an assignment operation (such as a=2)?
• Are separate copies of data created or references passed to the original when they are sent as function parameters?

In this article, I will summarize the findings by presenting a
comparative analysis. Read the entire article here.

Data in C

In C, the data you use in your programs will usually fall into one of the three
basic categories: int, char and float. Data in C has no
existence without an associated memory location labeled by an
identifier, usually referred to as a variable (the term variable
is a bit misleading, since it essentially means that it must always
vary, but you can have constant variables – i.e. variables whose
values do not vary). Considering this and C’s requirement for static
typing, a variable declaration statement is required before data
can be stored in a variable. This declaration statement usually takes the
form of data-type var-name [= value], where the =value part may
or may not be present. For example, the statement int a=1;
declares a variable a which will store integer data and stores
1 in it. What this statment basically tells the C compiler is
that it should allocate a block of memory large enough to store an
integer and it will referred to as a. It is possible to obtain the
address of this memory location using the & operator.

Read the entire article here.

PiCloud and REST API with C client

I was playing around with PiCloud’s REST API and had a bit of fun writing a C client for it. Now, why C? Well, because one of the reasons they provide a REST API is for your non-Python programs to easily access your Python code on PiCloud.

What I did is simple: I published the square_func using PiCloud’s documentation. Then I used libcurl’s C Interface to replicate the curl statements provided in the documentation. I have
written the client in two parts: client1.c and client2.c. client1.c invokes the published function, hence gets the Job ID. client2.c then uses this jobID to get the result. Here are the source codes:

/*client1.c*/
/* This part of the client invokes the REST API of
   PiCloud and retrieves the Job ID
   http://docs.picloud.com/rest.html#invoking-functions
*/

#include <stdio.h>
#include <curl/curl.h>

int main(void)
{
  CURL *curl;

  /* Make sure you set this appropriately*/
  char *url="https://api.picloud.com/r/3222/square_func/";
  CURLcode res;
  
  curl = curl_easy_init();
  if(curl) {
    /* First set the URL that is about to receive our POST. This URL can
       just as well be a https:// URL if that is what should receive the
       data. */
    curl_easy_setopt(curl, CURLOPT_URL, url);
    /* Specify the user/pass */
    curl_easy_setopt(curl,CURLOPT_USERPWD,"apikey:secretkey");
    
    /* Now specify the POST data */
    curl_easy_setopt(curl, CURLOPT_POSTFIELDS, "x=5");
    
    /* For HTTPS */
    curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 0L);

    /* Perform the request, res will get the return code */
    res = curl_easy_perform(curl);
    printf("\nResult of Operation:: %d\n", res);

    /* always cleanup */
    curl_easy_cleanup(curl);
  }
  return 0;
}

/*client2.c*/
/* This part of the client retrieves the result given the
   job ID as the argument*/
#include <stdio.h>
#include <stdlib.h>
#include <curl/curl.h>

int main(int argc, char *argv[])
{
  CURL *curl;
  char url[80];
  CURLcode res;

  if (argc==1)
    {
      printf("Usage: ./client2 <jid>\n");
      exit(0);
    }

  
  strcpy(url,"https://api.picloud.com/job/result/?jid=");
  strcat(url,argv[1]);

  

  curl = curl_easy_init();
  if(curl) {
    /* First set the URL that is about to receive our POST. This URL can
       just as well be a https:// URL if that is what should receive the
       data. */
    curl_easy_setopt(curl, CURLOPT_URL, url);
    
    /* Specify the user/pass */
    curl_easy_setopt(curl,CURLOPT_USERPWD,"apikey:secretkey");
    
    /* for HTTPS */
    curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 0L);
    
    /* Perform the request, res will get the return code */
    res = curl_easy_perform(curl);
    printf("\nResult of Operation:: %d\n", res);

    /* always cleanup */
    curl_easy_cleanup(curl);
  }
  return 0;
}

Working

[gene@zion tmp]$ ./client1
{“jid”: 57, “version”: “0.1”}
Result of Operation:: 0

[gene@zion tmp]$ ./client2 57
{“version”: “0.1”, “result”: 25}
Result of Operation:: 0

Hope it works for you. PiCloud is fun!