How to Reduce Azure VM Cost?

Are you frustrated with increasing Azure cloud costs? For most Azure workloads, the biggest factor affecting the cost is the Azure VM resource. However, it is possible to reduce VM cost by adopting a set of strategies in provisioning and using the VM resources. This article provides 10 tips you can use today to reduce your Azure VM cost.

Overview Azure VM Cost Reduction Tips

Here are the top 10 tips for reducing Azure VM cost,

  1. Select the Right VM Type for your Workload
  2. Select the Right VM Size for your Workload
  3. Use a Low Cost Region for your VMs
  4. Use Azure Spot Instance
  5. Use Azure Hybrid Benefit
  6. Use Reserved Azure VM Instance
  7. Reduce Cost by Using a Smaller VM Disk
  8. Use Auto Shutdown Feature for VM
  9. Use Azure Advisor to Monitor Cost Reduction Opportunities
  10. Analyse your VM Cost Breakdown for Optimising Cost

Details on each of the Azure VM cost reduction tips is given below,

1. Select the Right VM Type for your Workload

There are 3 key decisions you need to make before you decide to create an Azure VM. This includes the type of VM you need, the type of OS you want to run on the VM and the region where you want to run your VM. Note that the available options for VMs change based on your region and OS requirements. Use the Azure VM selector tool to narrow down your VM choice.

After specifying the OS and your hosting region requirement (if any), the most important thing to decide is the VM type based on your workload requirement. At a high level, VM types for the following common workload types are available,

  • General-purpose workloads
  • Compute-intensive workloads
  • Storage-intensive workloads
  • Memory-intensive workloads
  • GPU-enabled workloads
  • High-performance computing (HPC) workloads

Once the VM type is selected, you can fine tune the VM selection by using the following requirement filters,

  • The CPU brand you need - AMD, Intel or Ampere Altra arm
  • Type of CPU usage - continuous or burst usage
  • Do you need a GPU in the VM?
  • Do you need a storage or data disk attached?

According to this Microsoft blog, by using the newer Ampere Altra arm CPU, you can get up to 50% better price-performance for your cloud workloads!

Here is a sample VM type selection - A memory optimised VM instance in the West Europe region running CentOS 7.9 under AMD CPU with continuous usage and no data disk.

2. Select the Right VM Size for your Workload

After selecting the Azure VM type, the next major decision affecting the cost is the size of the VM. This is a combination of vCPU cores, memory, I/O speed and disk specifications. It is important to choose the lowest configuration that meets your workload requirements. This typically involves starting with a higher VM size, monitoring the key metrics of the provisioned VM and then optimising for the right VM size. Another approach will be to start with the smallest size and then upgrade as per workload requirement.

Here is a sample VM size selection - A memory optimised VM instance in the West Europe region running CentOS 7.9.  Since the workload require at least 32GB, the initial VM model selected is E4bds_V5.

Azure VM Size Selection

3. Use a Low Cost Region for your VMs

For most workloads, you will require a specific region for your VMs. However sometimes you may not need a specific region. In such cases, ensure that you provision your VM in a region which offers the lowest cost for your VM. For example, the East US region usually has 25% cheaper VM prices than the West Europe region. See this page for a region level price comparison of Azure VMs.

It is very important in this case to ensure that you are not doing any large scale data transfers between your VM in a separate region and your other resources in another region as it will lead to bandwidth cost and slow performance. Also please be aware of compliance and legal requirements when it comes to hosting your VM workloads in a specific region.

4. Use Azure Spot Instance

When you provision a VM, you have an option to specify it as a Spot VM. Spot VMs are up to 85% cheaper than usual VM instances since Azure uses its free available data centre capacity to power them. However, the catch is that your VM can be shutdown abruptly if Azure runs out of free VM instance pool. Hence this is only useful if you want to run some of your workloads using random available instances.

Clearly spot instances are not useful for always on services such as database servers or web applications. However, it is a good choice if you want some kind of background processing that can take its own sweet time to complete.

Azure spot instance options

5. Use Azure Hybrid Benefit

If you are provisioning Windows server based VMs on Azure, you can save up to 49% by using your own Windows server licences. This hybrid benefit can be selected during the creation of Windows server VMs.

Azure hybrid benefit

6. Use Reserved Azure VM Instance

Azure provides 4 different pricing options for virtual machines. These options ordered from high cost to low cost are pay as you go, 1 year reserved instances, 3 year reserved instances and spot instances. Please note that some options are available only to Azure customers in specific regions. For example, VM reservation for 1 year or 3 years is not available for Indian customers. Following are monthly prices for a sample VM for each option in the West Europe region (since prices also depend on the region). VM Type selected is Standard D4s V3 - 4vCPU and 16GB,

  • Pay as you go option - $175
  • 1 year VM reservation - $152 (13% discount)
  • 3 years VM reservation - $120 (31% discount)
  • Azure spot instance - $26 (85% discount)

If you are using a large pool of VMs, it is better to use a mixed reservation approach - you can reserve some for 3 years, some for short term 1 year and keep some unreserved if you plan to do some optimisations to reduce VM instances. See this page for various Azure VM pricing options.

7. Use Auto Shutdown Feature for VM

Azure has a simple shutdown feature which can be configured to enable auto shutdown of a virtual machine at a specified time. This is quite useful for machines that are needed during work hours only. Even if you forget to switch off the VM after your work, the auto shutdown can kick in and turn off the machine. Auto shutdown feature is available for a VM under the Operations section on the left of the VM configuration page.

Azure auto shutdown

Sometimes you want your VM to be running for a specific time duration only. For example, you may need your machine running only between 9AM and 11AM. In such cases you can set up automation tasks for starting and stopping VMs. Use the Tasks link under the Automation section available on the left of the VM configuration page.

Azure auto startup

8. Reduce Cost by Using a Smaller VM OS Disk

When an Azure VM instance is created using the portal, you can only specify the type of disk (HDD or SSD) but not the size of the disk. Many OS images create disks of 128GB size and these also add to the cost of the VM. Hence one way to reduce the cost is to use a lower size OS disk. However, this option is not available on the portal, and hence you need to use PowerShell command or Azure CLI command to use a custom sized OS disk. Alternatively, you can create the VM from the portal and then reduce the OS disk size using the CLI.

9. Use Azure Advisor to Monitor Cost Reduction Opportunities

When you have a large number of resources under your Azure tenant, it is easy to miss cost optimisation options. In such cases, it is important to monitor cost reduction recommendations provided by Azure Advisor. Azure advisor has a separate section on cost recommendations and is capable of providing VM cost recommendations such as reservation options and reduction of VM size based on historic CPU usage. If you are part of a large organisation with a large set of Azure resources, it is important to have a weekly monitoring cost KPI driven by Azure cost Advisor.

Azure cost advisor

10. Analyse your VM Cost Breakdown for Optimising Cost

An Azure VM provisioned usually has the following additional infrastructural components.

  • OS Disk and Data disk attached to the VM
  • Any public IP attached to the VM
  • Networking traffic on VM network interface such as cost of network peering

Each of these components can incur additional cost over and above the VM cost. So it is recommended to analyse the cost incurred by these VM components in Azure portal periodically to further optimise VM cost.

GraalVM What is it?

Features of GraalVM

GraalVM is a high performance JDK with a focus on 3 core features - high performance, compiled native app and polyglot programming. It is a drop-in replacement for Java 11 and Java 17. GraalVM community edition is an open source project with over 3.6 million lines of code contributed by the GraalVM team and collaborators.

GraalVM adds the following things to the Java HotSpot VM,

  • A GraalVM compiler that is capable of compiling applications to a native image that is built for the underlying operating system. This ensures high performance and quick to start applications suitable for cloud and container platforms. With static linking of system libraries, a static binary can be even run on a bare-metal docker image!
  • A language implementation framework codenamed Truffle that makes it possible to run programming languages such as JavaScript, Ruby, Python etc. on top of JVM. Ability to interoperate across languages under a shared runtime makes GraalVM a true polyglot programming environment.
  • GraalVM LLVM runtime that can execute programming languages that can be converted to LLVM bitcode. This includes languages such as C, C++ and Fortran.
  • A low overhead language agnostic fully dynamic instrumentation support available as part of the VM runtime. This can be used by API clients. It can track very fine-grained VM-level runtime events to inspect, profile and analyse the runtime behaviour of applications running on GraalVM.

The key vision statement of GraalVM is,

"Abstractions should be without performance regret!"

As per Oracle’s initial announcement, GraalVM can provide up to 100x faster startup time and up to 5x reduction in memory usage by using the ahead-of-time native compilation. This makes it a very tempting choice for building cloud native microservices.

GraalVM Architecture Diagram

The following diagram shows the high level layered architecture of the GraalVM ecosystem supporting native image compilation and polyglot features,

GraalVM Architecture Diagram

GraalVM is a project managed by Oracle and provides both enterprise and community edition distributions.

Goals of GraalVM Project

  • Improve the performance of Java VM languages to match the performance of native languages and ensure low memory footprint.
  • Reduce the startup time of JVM applications by compiling them ahead-of-time to native code.
  • Enable GraalVM to be an embeddable technology, for example inside Oracle database.
  • Allow use of multiple programming languages in a single program, the polyglot way.
  • Enable application extensibility by supporting reuse of libraries from other languages or building language agnostic tools.

A Brief History of GraalVM

The word Graal comes from the old French for Grail (a thing to be eagerly pursued). Graal project started as a Java compiler research project inside Oracle Labs under the leadership of Thomas Wuerthinger. The original goal was to create a faster compiler for Java but then evolved to build additional features. These included ahead-of-time compilation called native image technology supporting fast native binaries and a language implementation framework enabling polyglot capabilities using non-JVM languages such as Python, Ruby and WebAssembly. All the technologies under Graal were released under the single umbrella name GraalVM.

The GraalVM ecosystem was in development for years in Oracle labs. In June 2018, a release candidate (1.0-RC2) version was released under free community edition. The first production ready version was released by Oracle under GraalVM enterprise edition 19.0 in May 2019 as a drop-in replacement for JDK 8. In October 2022, Oracle announced GraalVM JIT and native image becoming part of OpenJDK.

Following shows the evolution of GraalVM from May 2019 to October 2022. These versions are available either in community edition or enterprise edition with enterprise edition having additional features.

GraalVM 19.x versions (May 2019 to April 2021)

  • Added support for JDK 11
  • Improvements to GraalVM compiler
  • Upgrades to Truffle language runtimes

GraalVM 20.0.x to GraalVM 20.3.x (February 2020 - January 2022)

  • Full support for Windows platform
  • Improved native images
  • Upgrades to Truffle language runtimes
  • Improved tooling support

GraalVM 21.0.x to GraalVM 21.3.x (January 2021 - September 2022)

  • Java on Truffle
  • GraalVM community images from Github container registry
  • Linux ARM 64-bit support
  • Serialisation support for images
  • Tooling and language runtime updates
  • Java 17 support

GraalVM 22.0.x to GraalVM 22.3.x (January 2021 - October 2022)

  • Dropped support for Java 8. Now supports JDK 11 and 17 only
  • Native image optimizations reduced executable size
  • Updated to Truffle language implementations
  • Added support for Apple silicon ARM M1

Limitations and Disadvantages of GraalVM

There is a lot of hype in the Java ecosystem about GraalVM. However, it is important to be aware of the limitations and disadvantages of GraalVM before adopting it. The ability of GraalVM to create native executables using SubstrateVM while exciting also comes with certain limitations. It is important to know these before deciding to adopt native images as your solution format,

  • Java can use reflection to dynamically load classes. This is a problem for native compiler since it needs to know all the classes that may be loaded at runtime to generate the native image. The workaround is to specify all the classes needed using the native image configuration files.
  • A lot of third party libraries and frameworks use reflection heavily. Hence those need to support GraalVM explicitly for it to work. See Spring Native.
  • The native executables created do not have a JIT compiler. Hence aggressive code optimisation at runtime using profiler and JIT is not possible.
  • Community edition of GraalVM provides a serial garbage collector as part of the native image runtime. It is intended for low memory footprint and small Java heap sizes. The runtime behaviour in this case can be different from a JVM based GC.

Language and Runtime Support in GraalVM

GraalVM is designed for the Java ecosystem and hence can execute all languages that compile to the Java bytecode format. These include languages such as Java, Scala, Kotlin etc.

In addition, the Truffle Language Implementation Framework supports the following additional languages,

  • JavaScript - ECMA 2021 JavaScript runtime with NodeJS support
  • Ruby - Ruby language implementation with support for Ruby on Rails
  • R - R language implementation
  • Python - Python 3 language implementation
  • WASM - WebAssembly implementation
  • LLVM - Sulong LLVM bitcode interpreter implementation

Who Uses GraalVM in Production?

Due to GraalVM’s diverse toolset and cloud advantages, some of the big players in the tech industry are currently using GraalVM in production. These include Facebook, Oracle, NVidia, Twitter, Goldman Sachs etc. Some of the examples of GraalVM in production are given below,

  • Facebook with Spark workloads - Facebook has huge data sets and Java is used in big data processing using technologies such as Spark. By adopting GraalVM, Facebook was able to get reduced CPU and memory utilisation for spark workloads. Facebook using GraalVM for more details.
  • Twitter Microservices Platform - Most of Twitter’s microservices are written in JVM language Scala. Using the GraalVM compiler, the CPU consumption was reduced more than 10%. The key technical features behind this improvement are better code inlining and escape analysis by the new compiler. Twitter using GraalVM in production.
  • Goldman Sachs Slang Language Migration - In this interesting use case, Goldman Sachs is using the GraalVM’s Truffle language implementation framework to modernise their in-house programming language called Slang (Securities Language) with ability to interoperate with legacy C++ code. Team found tooling, native interoperate and documentation as key good parts of the GraalVM ecosystem. GoldMan Sachs Using GraalVM in Production for more details.

What You Can Do with GraalVM?

  • Use it as a platform for high performance modern Java.
  • Use it in container or cloud environments as a low footprint and fast startup Java microservices.
  • Use multiple languages such as Java, JavaScript, Ruby etc. for a single application.
  • Run native languages such as C, C++ on the JVM.
  • Create tools that work across all languages.
  • Extend a JVM based application.
  • Extend a native application.
  • Build Java code as a native library.
  • Implement polyglot features in database.
  • Create your own language.

GraalVM - Additional Reading

Recommended Books for Beginning Python Programmers

Python is currently the most popular programming language. Due to its simplicity and expressive power it is being used across the world as the first language to teach programming. It is also now used in advanced programming use cases such as data science, machine learning and statistics. With a powerful and large library of code, applications can be quickly built using Python.

For a beginning programmer starting to learn python, the first recommended resource is the official homepage of Python language itself. The beginner's guide to python provides a guided tour of the language, libraries and various use cases. . If python is your first language, start from here or otherwise start from this page.

Another way to learn python is to start with a book that provides a complete overview of the language and basics of using python. In this article, I will provide a set of book recommendations and what you will find in these books. Some folks like starting with a book than online documentation since it provides direct and focused approach to learning.

A Byte of Python by Swaroop CH - This is a free book on python available online. A PDF copy is also available for download. It is a gentle introduction to python and quickly introduces you to the python ecosystem and fundamentals. It doesn't cover much about programming in general, but as a first introduction to the python language, I recommend this book. The book is around 100 pages and doesn't go into advanced features of python. The book covers installing and getting started, basics, operators, control flow, functions, modules, basic data structures, basics of object oriented programming/problem solving, input/output, exceptions and standard library. This book is available as an open source project here.

Python Crash Course 2nd Edition by Eric Matthes - If you can spend some money, Python crash course is my recommended book for beginners. This book takes a project based approach to teaching python. The first section of the book is a good introduction of the python language starting from basics to object oriented programming using classes. This section is full of code snippets and try it yourself programming problems that gets you quick into the practical usage of python language. The second section shows you how you can build projects using python. It covers 3 major projects consisting of building a game, data visualization and creating web applications using python. The appendix covers installation, use of IDEs and a brief intro to the git version control. All in all, best book for python programmers at the beginner level. With over 500 pages, it may take a while cover the entire material.

Once you have completed these beginner books and has practical experience working with python projects, you may want to learn a bit more about pythonic way of writing code. Following are my recommendations for intermediate to advanced python programmers,

Effective Python by Bret Slatkin - Effective python is a mandatory reading for anyone who uses python in a professional setting or uses it frequently. It consists of 90 important things you need to know when writing python programming in python language organized in 10 chapters. Based on the style of classic effective c++ by Scott Meyers, this is a must have book for any programmer's library. Each advice consists of a brief background followed by plenty of code examples illustrating how the advice is to be applied when writing python code. Reading this book makes you understand how expert programmers write python code. Some examples from the 90 items included in the book are - Prefer raising exceptions to returning None (item 20), Profile before optimizing (item 70) and Consider interactive debugging with pdb (item 80).

Fluent Python by Luciano Ramalho - Fluent python is an advanced book on python that will enable you to understand and write effective and idiomatic python code. As the author himself says in preface, this book was written for practicing Python programmers who want to become proficient in Python 3. With over 700 pages, this book covers python data model, data structures, object oriented idioms, advanced control flows such as generators and coroutines and meta programming. It covers almost all advanced python features an experienced programmer is expected to know such as design patterns, function decorators and closures, ABC, inheritance, operator overloading, context managers, concurrency, class metaprogramming etc.

Python Cookbook by David Beazley and Brian K Jones - David Beazley is a well known software engineer who has made substantial contributions to the python developer community. His video sessions and keynotes are available on youtube and they are notable for the in-depth look at python and its ecosystem. Python cookbook is intended for experienced python programmers looking to deepen their understanding of the language and advanced programming techniques and idioms in python. This book organizes python recipes/techniques under 15 different chapters covering areas from data structures/algorithms, data types(string,numbers, dates etc.), modules, metaprogramming, network programming, c extensions etc. Each recipe is organized in 3 sections - problem, solution and discussion. Some example recipes are sanitizing and cleaning up text, reading and writing binary data and replacing single method classes with functions.

Python in a Nutshell by Alex Martelli- Alex Martelli is a software engineer and fellow of the python software foundation. This book is intended for programmers with some python experience. It is organized in 5 sections - getting started with python, core python language and built-ins, python library and extension modules, network and web programming, extending/distributing and v2/v3 migration. This is an excellent reference to python core and provides an in-depth look at language. However some may find it difficult due to the extreme detailing of the language (book is 700+ pages!).

How to Generate QR Codes Using Python

What is a QR Code?

Sample QR CodeA QR code (Quick Response code) is a two dimensional barcode used popularly in product tracking, item recognition, document tracking and general marketing. It is also now commonly used in entry tickets since it can be easily scanned using a QR code scanner in a mobile. Standard QR codes can contain upto 3Kb of data. See this page to know how a QR code stores data.

QR codes may be used to display text, encode a web page URL etc. In the following python example, we will encode the URL of this website. Hence by scanning the resulting QR code image, you can navigate to www.quickprogrammingtips.com.

Generating QR Code Using Python

The following example uses pyqrcode module available from python repository. It is an easy to use module written in pure python. Optionally you can use it along with pypng module to render png files.

Before writing and running the following sample python program, ensure you have pyqrcode installed by running the pip command,

pip3 install pyqrcode

If you plan to create png files as qr code output, install pypng as well,

pip3 install pypng

Let us now quickly create a simple QR code which contains the data, "www.quickprogrammingtips". By scanning the QR code, you can visit this site. Type the following code in a file named qrcodegenerator.py,

# Import pyqrcode module. Ensure it is installed using pip3 install pyqrcode
import pyqrcode
# Import png module. Ensure it is installed using pip3 install pypng.
import png
from pyqrcode import QRCode


data = "www.quickprogrammingtips.com"

# create qr code using the alphanumeric encoding of the data (url string in this case)
qrobject = pyqrcode.create(data)

# Create and save svg and png files
qrobject.svg("siteqrcode.svg", scale = 5)
qrobject.png('siteqrcode5.png', scale = 5)
qrobject.png('siteqrcode1.png', scale = 1)
qrobject.png('siteqrcode10.png', scale = 10)

Run the above code using,

python3 qrcodegenerator.py

The above code generate qr codes in svg and png format in the local folder. We have also created 3 png files with different scale parameters. A scale value of 1 defines the data module size of 1 pixel and hence will be too small. A scale factor of 5 or more creates larger image sizes that can be easily scanned. With a scale factor of 5, qr code images created was around 200px in width and height.

The above code used a simple version of the pyqrcode.create() function. For advanced use it can take additional parameters such as error level, qr version and mode. A typical use case will be - pyqrcode.create('hello', error='L', version=27, mode='binary'). Let me explain what these additional parameters are,

  • Error level - QR codes can use one of four possible error correction values. They are referred to by the letters: L, M, Q, and H. The L error correction level corresponds to 7% of the code can be corrected. The M error correction level corresponds to 15% of the code can be corrected. The Q error correction level corresponds to 25% of the code can be corrected. The H error correction level corresponds to 30% of the code can be corrected.
  • Version - QR codes have versions 1 to 40. Higher version can hold more data with increased number of dots. When not specified, lowest viable version will be selected based on the data encoding and data correction level.
  • Mode - The encoding used to represent the data in a QR code. There are four possible encodings: binary, numeric, alphanumeric, kanji. When not specified, the library will pick one based on the data passed.

The following example creates a QR code for the text "Hello World" using the advanced parameters.

# Import pyqrcode module. Ensure it is installed using pip3 install pyqrcode
import pyqrcode
# Import png module. Ensure it is installed using pip3 install pypng.
import png
from pyqrcode import QRCode


data = 'Hello World'

# create qr code using the alphanumeric encoding of the data
# Since the data contains a space
qrobject = pyqrcode.create(data, error='H', version=10, mode='binary')

# Create and save svg and png files
qrobject.png('messageeqrcode5.png', scale = 5)

If you specify a lower version number that cannot hold the data you want to encode, you will get the follwing error,

The data will not fit inside a version code with the given encoding and error level.

How to Add or Remove Columns from Sqlite Tables

Sqlite is a lightweight and powerful database engine. It is used mobile platforms such as Android and iOS as the default database engine. In many of my Android applications, I use pre-populated sqlite databases for data that changes rarely. As I released new versions of the app, sometimes I had to add, remove or rename columns of sqlite tables with data. Interestingly depending on the version of sqlite used in your machine, there are multiple ways to alter columns of sqlite tables.

If you are using sqlite 3.35 or later you can use the following commands,

  • Add a column - ALTER TABLE [table name] ADD COLUMN [column name]
  • Rename a column - ALTER TABLE [table name] RENAME COLUMN [column name]
  • Remove a column - ALTER TABLE [table name] DROP COLUMN [column name]

However if you are using sqlite versions between 3.25 and 3.35, you have access only to the following commands,

  • Add a column - ALTER TABLE [table name] ADD COLUMN [column name]
  • Rename a column - ALTER TABLE [table name] RENAME COLUMN [column name]

If you are using sqlite version 3.2, you have access only to the following command,

  • Add a column - ALTER TABLE [table name] ADD COLUMN [column name]

None of the above commands are available if you are using sqlite versions prior to 3.2. However you can use a multi-step process given below to migrate tables without loosing data,

  • Rename the current table
  • Create a new table with previous name
  • Copy data from renamed table to the new table
  • Drop the renamed table

Let me illustrate the various options by using a sample scenario. Assume we have a customer table which contains columns id, fullname and email.

CREATE TABLE customer(
 id VARCHAR(10) PRIMARY KEY,
 fullname VARCHAR(128),
 email VARCHAR(128));

Assume we want to drop the email column and then add an address column. In Sqlite 3.35 and above you can use the following commands,

ALTER TABLE customer ADD COLUMN address VARCHAR(128);
ALTER TABLE customer DROP COLUMN address;

In older versions, you can use the following generic approach.

1. Rename the current customer table,

ALTER TABLE customer RENAME TO old_customer;

2. Create a new customer table,

CREATE TABLE customer(
 id VARCHAR(10) PRIMARY KEY,
 fullname VARCHAR(128),
 address VARCHAR(128));

3. Copy data from old table (note that address field will be blank for all rows),

INSERT INTO customer SELECT id, fullname,"" FROM old_customer ;

4. Finally delete the old table,

DROP TABLE old_customer;

Alternatively you can use a tool such as Sqlite Browser to change database structure. In Sqlite browser, click on database structure and then select the table. Click modify table button on top. From the next window you can add, remove or modify order of columns.