Recommended Books for Beginning Python Programmers

Python is currently the most popular programming language. Due to its simplicity and expressive power it is being used across the world as the first language to teach programming. It is also now used in advanced programming use cases such as data science, machine learning and statistics. With a powerful and large library of code, applications can be quickly built using Python.

For a beginning programmer starting to learn python, the first recommended resource is the official homepage of Python language itself. The beginner's guide to python provides a guided tour of the language, libraries and various use cases. . If python is your first language, start from here or otherwise start from this page.

Another way to learn python is to start with a book that provides a complete overview of the language and basics of using python. In this article, I will provide a set of book recommendations and what you will find in these books. Some folks like starting with a book than online documentation since it provides direct and focused approach to learning.

A Byte of Python by Swaroop CH - This is a free book on python available online. A PDF copy is also available for download. It is a gentle introduction to python and quickly introduces you to the python ecosystem and fundamentals. It doesn't cover much about programming in general, but as a first introduction to the python language, I recommend this book. The book is around 100 pages and doesn't go into advanced features of python. The book covers installing and getting started, basics, operators, control flow, functions, modules, basic data structures, basics of object oriented programming/problem solving, input/output, exceptions and standard library. This book is available as an open source project here.

Python Crash Course 2nd Edition by Eric Matthes - If you can spend some money, Python crash course is my recommended book for beginners. This book takes a project based approach to teaching python. The first section of the book is a good introduction of the python language starting from basics to object oriented programming using classes. This section is full of code snippets and try it yourself programming problems that gets you quick into the practical usage of python language. The second section shows you how you can build projects using python. It covers 3 major projects consisting of building a game, data visualization and creating web applications using python. The appendix covers installation, use of IDEs and a brief intro to the git version control. All in all, best book for python programmers at the beginner level. With over 500 pages, it may take a while cover the entire material.

Once you have completed these beginner books and has practical experience working with python projects, you may want to learn a bit more about pythonic way of writing code. Following are my recommendations for intermediate to advanced python programmers,

Effective Python by Bret Slatkin - Effective python is a mandatory reading for anyone who uses python in a professional setting or uses it frequently. It consists of 90 important things you need to know when writing python programming in python language organized in 10 chapters. Based on the style of classic effective c++ by Scott Meyers, this is a must have book for any programmer's library. Each advice consists of a brief background followed by plenty of code examples illustrating how the advice is to be applied when writing python code. Reading this book makes you understand how expert programmers write python code. Some examples from the 90 items included in the book are - Prefer raising exceptions to returning None (item 20), Profile before optimizing (item 70) and Consider interactive debugging with pdb (item 80).

Fluent Python by Luciano Ramalho - Fluent python is an advanced book on python that will enable you to understand and write effective and idiomatic python code. As the author himself says in preface, this book was written for practicing Python programmers who want to become proficient in Python 3. With over 700 pages, this book covers python data model, data structures, object oriented idioms, advanced control flows such as generators and coroutines and meta programming. It covers almost all advanced python features an experienced programmer is expected to know such as design patterns, function decorators and closures, ABC, inheritance, operator overloading, context managers, concurrency, class metaprogramming etc.

Python Cookbook by David Beazley and Brian K Jones - David Beazley is a well known software engineer who has made substantial contributions to the python developer community. His video sessions and keynotes are available on youtube and they are notable for the in-depth look at python and its ecosystem. Python cookbook is intended for experienced python programmers looking to deepen their understanding of the language and advanced programming techniques and idioms in python. This book organizes python recipes/techniques under 15 different chapters covering areas from data structures/algorithms, data types(string,numbers, dates etc.), modules, metaprogramming, network programming, c extensions etc. Each recipe is organized in 3 sections - problem, solution and discussion. Some example recipes are sanitizing and cleaning up text, reading and writing binary data and replacing single method classes with functions.

Python in a Nutshell by Alex Martelli- Alex Martelli is a software engineer and fellow of the python software foundation. This book is intended for programmers with some python experience. It is organized in 5 sections - getting started with python, core python language and built-ins, python library and extension modules, network and web programming, extending/distributing and v2/v3 migration. This is an excellent reference to python core and provides an in-depth look at language. However some may find it difficult due to the extreme detailing of the language (book is 700+ pages!).

How to Generate QR Codes Using Python

What is a QR Code?

Sample QR CodeA QR code (Quick Response code) is a two dimensional barcode used popularly in product tracking, item recognition, document tracking and general marketing. It is also now commonly used in entry tickets since it can be easily scanned using a QR code scanner in a mobile. Standard QR codes can contain upto 3Kb of data. See this page to know how a QR code stores data.

QR codes may be used to display text, encode a web page URL etc. In the following python example, we will encode the URL of this website. Hence by scanning the resulting QR code image, you can navigate to www.quickprogrammingtips.com.

Generating QR Code Using Python

The following example uses pyqrcode module available from python repository. It is an easy to use module written in pure python. Optionally you can use it along with pypng module to render png files.

Before writing and running the following sample python program, ensure you have pyqrcode installed by running the pip command,

pip3 install pyqrcode

If you plan to create png files as qr code output, install pypng as well,

pip3 install pypng

Let us now quickly create a simple QR code which contains the data, "www.quickprogrammingtips". By scanning the QR code, you can visit this site. Type the following code in a file named qrcodegenerator.py,

# Import pyqrcode module. Ensure it is installed using pip3 install pyqrcode
import pyqrcode
# Import png module. Ensure it is installed using pip3 install pypng.
import png
from pyqrcode import QRCode


data = "www.quickprogrammingtips.com"

# create qr code using the alphanumeric encoding of the data (url string in this case)
qrobject = pyqrcode.create(data)

# Create and save svg and png files
qrobject.svg("siteqrcode.svg", scale = 5)
qrobject.png('siteqrcode5.png', scale = 5)
qrobject.png('siteqrcode1.png', scale = 1)
qrobject.png('siteqrcode10.png', scale = 10)

Run the above code using,

python3 qrcodegenerator.py

The above code generate qr codes in svg and png format in the local folder. We have also created 3 png files with different scale parameters. A scale value of 1 defines the data module size of 1 pixel and hence will be too small. A scale factor of 5 or more creates larger image sizes that can be easily scanned. With a scale factor of 5, qr code images created was around 200px in width and height.

The above code used a simple version of the pyqrcode.create() function. For advanced use it can take additional parameters such as error level, qr version and mode. A typical use case will be - pyqrcode.create('hello', error='L', version=27, mode='binary'). Let me explain what these additional parameters are,

  • Error level - QR codes can use one of four possible error correction values. They are referred to by the letters: L, M, Q, and H. The L error correction level corresponds to 7% of the code can be corrected. The M error correction level corresponds to 15% of the code can be corrected. The Q error correction level corresponds to 25% of the code can be corrected. The H error correction level corresponds to 30% of the code can be corrected.
  • Version - QR codes have versions 1 to 40. Higher version can hold more data with increased number of dots. When not specified, lowest viable version will be selected based on the data encoding and data correction level.
  • Mode - The encoding used to represent the data in a QR code. There are four possible encodings: binary, numeric, alphanumeric, kanji. When not specified, the library will pick one based on the data passed.

The following example creates a QR code for the text "Hello World" using the advanced parameters.

# Import pyqrcode module. Ensure it is installed using pip3 install pyqrcode
import pyqrcode
# Import png module. Ensure it is installed using pip3 install pypng.
import png
from pyqrcode import QRCode


data = 'Hello World'

# create qr code using the alphanumeric encoding of the data
# Since the data contains a space
qrobject = pyqrcode.create(data, error='H', version=10, mode='binary')

# Create and save svg and png files
qrobject.png('messageeqrcode5.png', scale = 5)

If you specify a lower version number that cannot hold the data you want to encode, you will get the follwing error,

The data will not fit inside a version code with the given encoding and error level.

How to Add or Remove Columns from Sqlite Tables

Sqlite is a lightweight and powerful database engine. It is used mobile platforms such as Android and iOS as the default database engine. In many of my Android applications, I use pre-populated sqlite databases for data that changes rarely. As I released new versions of the app, sometimes I had to add, remove or rename columns of sqlite tables with data. Interestingly depending on the version of sqlite used in your machine, there are multiple ways to alter columns of sqlite tables.

If you are using sqlite 3.35 or later you can use the following commands,

  • Add a column - ALTER TABLE [table name] ADD COLUMN [column name]
  • Rename a column - ALTER TABLE [table name] RENAME COLUMN [column name]
  • Remove a column - ALTER TABLE [table name] DROP COLUMN [column name]

However if you are using sqlite versions between 3.25 and 3.35, you have access only to the following commands,

  • Add a column - ALTER TABLE [table name] ADD COLUMN [column name]
  • Rename a column - ALTER TABLE [table name] RENAME COLUMN [column name]

If you are using sqlite version 3.2, you have access only to the following command,

  • Add a column - ALTER TABLE [table name] ADD COLUMN [column name]

None of the above commands are available if you are using sqlite versions prior to 3.2. However you can use a multi-step process given below to migrate tables without loosing data,

  • Rename the current table
  • Create a new table with previous name
  • Copy data from renamed table to the new table
  • Drop the renamed table

Let me illustrate the various options by using a sample scenario. Assume we have a customer table which contains columns id, fullname and email.

CREATE TABLE customer(
 id VARCHAR(10) PRIMARY KEY,
 fullname VARCHAR(128),
 email VARCHAR(128));

Assume we want to drop the email column and then add an address column. In Sqlite 3.35 and above you can use the following commands,

ALTER TABLE customer ADD COLUMN address VARCHAR(128);
ALTER TABLE customer DROP COLUMN address;

In older versions, you can use the following generic approach.

1. Rename the current customer table,

ALTER TABLE customer RENAME TO old_customer;

2. Create a new customer table,

CREATE TABLE customer(
 id VARCHAR(10) PRIMARY KEY,
 fullname VARCHAR(128),
 address VARCHAR(128));

3. Copy data from old table (note that address field will be blank for all rows),

INSERT INTO customer SELECT id, fullname,"" FROM old_customer ;

4. Finally delete the old table,

DROP TABLE old_customer;

Alternatively you can use a tool such as Sqlite Browser to change database structure. In Sqlite browser, click on database structure and then select the table. Click modify table button on top. From the next window you can add, remove or modify order of columns.

How to Capture TCP/IP Network Traffic From Kubernetes Pod

If you have applications running as containers in Kubernetes clusters, you may come across scenarios where you want to monitor or analyse raw network traffic. This is usually needed if your containers are making outbound API calls and you find random connection issues with your API calls. If you are using public cloud providers such as Azure, there are monitoring tools such as Network watcher which can be used to monitor traffic. However in this article I will show you a quick and easy way to monitor the pod network traffic at the TCP/IP level using pod console only.

We will be using a linux command line utility called tcpdump to capture TCP/IP network traffic. Note that API calls using http protocol actually runs on top of TCP/IP. Hence tcpdump provides a more precise view of networking issues at the TCP/IP level.

Step 1: Identify the pod name using the following command in you machine's command line. We will use the pod name to connect to the running pod. The following command assumes you already have kubectl client installed in your machine for accessing kubernetes clusters,

kubectl get pods

Step 2: Use the pod name returned by the above command to connect and get a direct command line access to the running pod's operating system. You need to replace POD_NAME with the name of the pod returned above,

kubectl exec POD_NAME -it -- sh

Step 3: Once you are connected to the pod, run tcpdump --version in the pod to check whether you have the tcpdump command available in your running pod. If not, run one of the following commands to install tcpdump to your running container. Please note that if your container is restarted any time, you will have to run the command again to get tcpdump installed.

If Debian based linux distributions is used in the pod, run follwing command to install tcpdump,

apt-get update && apt-get install tcpdump

If Alpine linux or busybox distributions is used in the pod, run following command to install tcpdump,

apk add tcpdump

Step 4: Now you are all set to capture network packets from the pod. Run the following command on the pod command line to capture network traffic to a file named networkcapture.cap. The -s option with 0 ensures capture of large network packets, -vvv option ensures detailed capture and -w specifies the file name for capture.

tcpdump -s 0 -vvv -w networkcapture.cap

The above command captures detailed logs for all TCP/IP transactions. If you want to limit the traffic to a destination IP or host name (when you are troubleshooting API call issues), you can run the following command,

tcpdump -s 0 -vvv -w networkcapture.cap host IP_OR_HOST

Step 5: When you want to stop the capture, you can press Ctrl-C to break out of tcpdump. Now you have networkcapture.cap available in your pod. Using pwd command find the location of the file and note it down. Use this value to replace REMOTE_POD_PATH_INCLUDING_FILE_NAME in the following command.

pwd

Step 6: Exit from the pod console using exit command. Run the following command from your machine to download the networkcapture.cap file from pod to your local folder.

kubectl cp POD_NAME:REMOTE_POD_PATH_INCLUDING_FILE_NAME LOCAL_PATH_INCLUDING_FILE_NAME

Here is how a sample command looks like,

kubectl cp myapp-5bbc4d64c5-6rsdx:/usr/docker/app/networkcapture.cap /Users/jj/networkcapture.cap

Step 7: You can perform a detailed analysis of the tcpdump capture using the wireshark tool. Download it from here.

MongoDB Date Comparison Query Examples

In this guide, I will show you various examples of date comparison based queries in MongoDB. If you use MongoDB in your projects, I highly recommend bookmarking this page as a quick reference for date comparison queries.

For the following MongoDB query examples, I will use a collection representing products created in an e-commerce system. Since the examples are focused on date comparison, our product collection will be a simple one consisting only of these fields - name, price, createddate, updateddate. By running the following command on MongoDB shell, we will insert few records into the product collection. Note that the date fields also contain time.

db.products.insert({ name: "PS5", 
    price: 400, createddate: ISODate("2022-01-10T08:30:00.000"), 
    updateddate: ISODate("2022-01-15T08:30:00.000")} )

db.products.insert( { name: "XBox X", 
    price: 320, createddate: ISODate("2022-02-15T18:30:00.000"), 
    updateddate: ISODate("2022-03-15T20:30:00.000")} )

db.products.insert( { name: "Nintendo Switch", 
    price: 220, createddate: ISODate("2022-02-10T14:30:00.000"), 
    updateddate: ISODate("2022-03-11T22:30:00.000")} )

MongoDB supports find() command for queries. Alternatively you can also use the powerful MongoDB aggregation pipeline for queries. For each query requirement, I will provide both find() query and aggregation pipeline query. However I recommend learning aggregation queries as it is much more powerful than find(). Also note that the following queries are tested on MongoDB version 4.4.

How to Find Documents Between Two Dates in MongoDB

The following query returns products created between 1st January 2022 and 10th February 2022. Note that in this case we are not interested in the time for the dates compared. Following is the find() query to find documents within 1st January 2022 and 10th February 2022,

db.getCollection('products').find(
    {createddate:{$gt:ISODate("2022-01-01"),
    $lt:ISODate("2022-02-10")}})

Following is the aggregate() pipeline query,

db.getCollection('products').aggregate(
    [{"$match" : {"createddate" : {"$gt":ISODate("2022-01-01"), 
    "$lt":ISODate("2022-02-10") } }}]);

The above queries return 1 record from the sample data we have added. Please note that when time is not specified, it is assumed to be 12AM UTC time and hence the record with date 2022-02-10T14:30:00.000 is not returned in the query.

The following find() and aggregation queries demonstrate the use of time in ISODate() and also how to filter queries with additional conditions. We want only PS5 products to be returned within the date range.

db.getCollection('products').find(
    {name:"PS5",
    createddate:{$gt:ISODate("2022-01-01T10:10:40"),
    $lt:ISODate("2022-02-10T22:20:01")}});
db.getCollection('products').aggregate([
    {"$match" : {"name":"PS5"}},
    {"$match" : {"createddate" : {"$gt":ISODate("2022-01-01"), 
    "$lt":ISODate("2022-02-10") } }}
]);

How to Find Documents Using Month or Year Comparison in MongoDB

Sometimes you may want to find all the documents where one of the date fields matches a specific month or year. Following aggregate query uses date expression operator to find all the products updated in the month of March. This query first adds a virtual field called month extracting the month field from the updateddate column. It is then used in the match pipeline.

db.getCollection('products').aggregate([
    {$addFields: {  "month" : {$month: '$updateddate'}}},
    {$match: { month: 3}}]);

Following find() query returns all products updated in the month of March,

db.getCollection('products').find({
  $expr: {
        "$eq": [{"$month": "$updateddate"},3]
  }
})

You can filter on the year field of updated date by replacing $month with $year.