Analogy between Windows (TM) and Cocaine!!!

Disclaimer: The views expressed in this post are my own and are not intended to harm anyone or to jeopardize interests of any company or organization, even Microsoft, since every company and individual has the right to protect their interests.

The Cocaine Analogy:

Imagine a govt making Cocaine or say any contraband drug available to anyone with the following condition:

1. Make the drug easily available Over the Counter from any medical store shop, legally, but at a very high price.

2.Make another channel for availability of drug, which is not legal, unofficial but known through word of mouth. The price should be substantially lower than the legal price and the quality of the drug, if you know what I mean, not as great as the legal one.

3. Make another channel for curbing the illegal sale of drug, say in the form of police. The intention is not to completely curb the  illegal sale of drug but only to curb it for those who are addicted too much to the drug.

In such a scenario, many people will first try out the drug, because it is easily available. Only when a particular product is easily available, even if not legal, ensures that the product has a larger user base. Over a period of time, they will get addicted. At the same time because of the curbing, people would be forced to buy the legal version but obviously at a higher price 🙂 Overall MS adopts the same strategy with almost all its products.

Analogy between Cocaine and Windows

To draw the analogy between Cocaine and Windows I list below what problem MS is trying to solve and what are the strategies that it adopts to solve those problems. For each strategy I mention what condition it fulfills in the cocaine analogy above :).

Problem: How to address piracy and at the same time make MS ‘Cocaine’ the most widely used platform.

Strategy:

Strategies fulfilling condition 1:

No strategy required. Simply make it available via partnered resellers.

Strategies fulfilling condition 2:

2.1 As soon a new version of MS ‘Cocaine’ is available, ensure that the Encryption Key for the ‘Cocaine’ is easily crackable / available.

2.2 Release a slightly modified version of ‘Cocaine’ (tagged as cracked with key / pirated) on various torrent websites.

This strategy fulfills the condition number 2 in the cocaine analogy.

Strategies fulfilling condition 3:

3.1 I prefer to write this as a pseudocode

os = check_for_installed_os
os = check_for_installed_os
type = check_user_type
if ( os == "pirated")
    if ( type == new )
        download_and_install_a_malicious_code_which_detects_user_behavior_and_submits_usage_information_to_MS
    else if (type == not_too_old)
        install_some_low_damaging_virus_partly_fixable_by_antivirus
        slowly_make_system_unusable
    else if (type == too_old)
        install_high_damage_virus_unfixable_by_any_antivirus
        immediately_make_system_unusable
else
    if ( type == too_old)
       virus_1 = download_low_damage_virus_from_ally_antivirus_company
       install virus_1
       virus_2 = download_low_damage_virus_from_MS_update
       install virus_2

       pester_user_to_buy_and_upgrade_antivirus_which_fixes_virus_1
       pester_user_to_buy_and_upgrade_to_the_newer_release_of_os_which_fixes_virus_2
   else
       inform_user_why_MS_cocaine_is_too_great
       suggest_user_to_buy_newer_cocaine_version
       suggest_user_to_buy_other_MS_products
       suggest_user_to_buy_antivirus_software

Other Strategies for making ‘Cocaine’ most widely used drug:

1. During Installation

while_installing_check_if_any_other_OS_except_cocaine*
remove_all_such_OS_installation
format_partitions_of_other_OS_installations_and_make_it_a_cocaine_partition

2. For programmers

2.1 Make all ‘Cocaine’ API specifications such that the code is unusable in any form; even copy_paste; for modifications when copied from ‘Cocaine’ to other platforms. Ensures that porting from ‘Cocaine’ to other platforms is too costly.

2.2 For big h/w companies: Pay them / work with them to make drivers for ‘Cocaine’.

2.3 For small h/w companies: Ask them to pay to provide support for writing drivers for ‘Cocaine’.

2.4 For big s/w companies: Ask them to pay to provide support for writing software for ‘Cocaine’.

2.5 For small s/w companies: DO_NOT_CARE

Note to Readers: Cocaine is a highly addictive substance. Please stop using Cocaine if you are addictive to it otherwise you might start showing ‘Cocaine bugs’ symptoms very soon. Chances are high that you are already addicted to Cocaine. In such a case, please visit a rehabilitation center ASAP.

I hope you enjoyed the read 🙂

Building Open Source libraries with Android NDK

Having scrambled through a NDK documentation and a lot of hit and trials and experimentation, finally I could figure out how could one build (even though partially) Open Source libraries with Android NDK.

Some background on how Autotools work:

The way autootool and friends work is this:

You write a configure.ac file, which is read in by Autoreconf and generates a configure script. Configure script test the build system for headers, libraries etc. and generates a config.h which contains definitions like #define HAVE_XXX 1 and also generates Makefiles after reading in the Makefile.in files which in turn is generated using Automake by reading in Makefile.am.

configure.ac —> input to –> autoreconf –> generates -> configure script –> checks host system for headers, libraries etc. -> generates -> config.h

Also

Makefile.am –> input to automake –> generates –> Makefile.in –> input to –> configure script –> generates Makefile

The code uses the generated config.h in the following fashion:

#ifdef HAVE_XXXX
#include <some-header>
#endif
….
….
#ifdef HAVE_XXXX
Call some function which is declared in the header
#else
Provide some other mechanism or report error to user or do whatever you want
#endif

The problem?

Most Open Source libraries use GNU autotools and its friends for building.

The first problem is that the Autools generate some configuration headers based on build time probe of the system. By build time probe I mean checking for things like if a header, library or a tool is present or not. In cross compiling scenario some of these probe should be done on the target system and not on the build system.

Second, the build system for most cross compiler tools have their own quirks which need passing some extra flags.

Third, in case of Android, it provides its own build system which are essentially some Makefile definitions and rules. This is provided so that people can build their code easily without having to deal with usual cross compiling issues. Thus there is a gap that while your autotools would generate the Makefiles while Android build system requires its own styled Makefiles in the form of Android.mk. One cannot simply generate Android.mk files using autotools.

Fourth, even if one gets to write Android specific Makefiles, the build would most probably fail as during the build it would look for a file config.h included in the fashion shown below, while no such file would exist as it is generated by the configure script.

#ifdef HAVE_CONFIG_H
#include “config.h”
#endif

Simply copying a config.h file from a run of configure script on another build system wouldn’t really work as the header files and other libraries present on Android may not match with the header and libraries present on the build system. Thus config.h would probably differ if it is somehow generated for Android with the one generated on a build system.

So how does one build an open source library for Android?

Solution:

The way I have managed to work around this trouble is to run the configure script with right cross compilation variables so that a config.h matching my Android system gets built and then writing Android.mk files which would simply use the Android build system.

The way I figured out the right flags was by building the Android source tree which displayed what flags are being used for building and then taking the cues from there, I passed the right flags to the configure script.

It looks something like this for building Android-3 target API on a linux host:

export ANDROID_ROOT=/home/divkis01/mydroid

The command above is sets the path where the Android sources are checked out from git respository.

NOTE: It is not necessary to check out the ANDROID sources and you can replace the ANDROID_ROOT with NDK_ROOT in all the commands below, along with proper path to the NDK cross compiler.

export PATH=$PATH:$ANDROID_ROOT/prebuilt/linux-x86/toolchain/arm-eabi-4.2.1/bin/

The command above is necessary so that configure can find out the path to the cross compiler which is needed to build some test programs during the configure run process. Please note that you can also set the path to the NDK compiler root

./configure –host=arm-eabi CC=arm-eabi-gcc CPPFLAGS=”-I$ANDROID_ROOT/build/platforms/android-3/arch-arm/usr/include/” CFLAGS=”-nostdlib” LDFLAGS=”-Wl,-rpath-link=$ANDROID_ROOT/build/platforms/android-3/arch-arm/usr/lib/ -L$ANDROID_ROOT/build/platforms/android-3/arch-arm/usr/lib/” LIBS=”-lc “

The command above has several points that should be well understood.

–host=arm-eabi –> This tells the configure script if the cross compilation is being done or not. It is also used as a prefix to some of the cross compiler tools like strip, nm etc.

CC=arm-eabi-gcc –> This tells the compiler that should be used for building

CPPFLAGS –> This tells the location where the header files should be searched which were specified in configure.ac with AC_CHECK_HEADER macro

CFLAGS=”-nostdlib” passes the option to build some test programs during configure process run. If you don’t pass this the compiler would link the standard C library of the host system which wouldn’t be compatible with the C library of the target system. You will end up getting error something like this, if you don’t pass this option:

/home/divkis01/mydroid/prebuilt/linux-x86/toolchain/arm-eabi-4.2.1/bin/../lib/gcc/arm-eabi/4.2.1/../../../../arm-eabi/bin/ld: crt0.o: No such file: No such file or directory

LIBS=”-lc” –> This option tells that it should explicitly link to a library called libc.so which is present in the location specified using the -L in the LDFLAGS option. If you are wondering that usually to build a C executable one doesn’t need to provide -lc as libc is automatically linked, then why do we need to specify this here? The answer lies in -nostdlib flag, which instructs not to link with the standard C library on the build system.

You will end up getting error something like this, if you don’t pass this option:

/home/divkis01/mydroid/prebuilt/linux-x86/toolchain/arm-eabi-4.2.1/bin/../lib/gcc/arm-eabi/4.2.1/../../../../arm-eabi/bin/ld: crt0.o: No such file: No such file or directory
collect2: ld returned 1 exit status

LDFLAGS = –> This option is also passed to build some test programs during configure process run.If you don’t pass the -Wl,-rpath-link option, then linker does not know where do the libraries dependent on the library specific using LIBS reside. If you don’t pass the -L option then the linker doesn’t know where do the libraries specified in LIBS reside.

You will end up getting error something like this, if you don’t pass the -Wl,-rpath-link option:

/home/divkis01/mydroid/prebuilt/linux-x86/toolchain/arm-eabi-4.2.1/bin/../lib/gcc/arm-eabi/4.2.1/../../../../arm-eabi/bin/ld: warning: libdl.so, needed by /home/divkis01/mydroid/development/ndk/build/platforms/android-3/arch-arm/usr/lib//libc.so, not found (try using -rpath or -rpath-link)
/home/divkis01/mydroid/prebuilt/linux-x86/toolchain/arm-eabi-4.2.1/bin/../lib/gcc/arm-eabi/4.2.1/../../../../arm-eabi/bin/ld: warning: cannot find entry symbol _start; defaulting to 00008184
/home/divkis01/mydroid/development/ndk/build/platforms/android-3/arch-arm/usr/lib//libc.so: undefined reference to `dl_unwind_find_exidx’

You will end up getting error something like this, if you don’t pass the -L option:

/home/divkis01/mydroid/prebuilt/linux-x86/toolchain/arm-eabi-4.2.1/bin/../lib/gcc/arm-eabi/4.2.1/../../../../arm-
eabi/bin/ld: cannot find -lc
collect2: ld returned 1 exit status

Once you run the configure script with these flags and options, it will generate the appropriate config.h which is compatible / in sync with your target system. Now you can go ahead and start writing the Android.mk files to build your sources.

Other troubles:

This solves only part of the problem as Autotools not only help in building but also in installing. Given it doesn’t make sense to install the build for a target system on host except for the headers. This can be done by augmenting the Android.mk files or writing some shell scripts to do this manually.

Conclusion:

Autotool is good only on GNU systems and using it for cross compiling can be really tedious, confusing, error prone or even impossible. The method described here is a hack and should be used at your own risk.

Let me know if this post was helpful for you.

CC=arm-eabi-gcc

Why debian rocks

It has been quite some time, actually years, since I started using Debian. Earlier I have used various flavors of Linux like SuSE, Red Hat, RHEL, Fedora, Ubuntoo but my experience with Debian has been so far the best. I just thought to write the reasons for which I really like Debian compared to other systems.

1. Excellent package management tools

I find that the package management tools that Debian offers is really fantastic. There are host of tools like Synaptic, Aptitude, Apt, dpkg, dselect etc. which one can choose based on how much one is comfortable with command line, TUI or GUI. One can easily choose and configure from where to install the package i.e. CD, standard Debian repositories hosted around the world, or some special Debian package repositories. Given I have my Debian CDs downloaded in ISO format, I can choose to loop mount them and install packages from them without the need to put the CDs for installation every time into the CDROM and without the need to download the package from internet every time. Above all the package dependencies are automatically resolved and installed / removed making the installation process a real breeze.

Apart from this there are host of packages like checkinstall (although available on other systems as well), localepurge, module assistant, debootstrap, pbuilder etc., which work so well in conjugation with other Debian package management tools that it is hard to describe in just one line or paragraph. Each of these tools really deserve a separate blog entry 🙂

In the past I have had real troubles specially with RPMS and local self installed packages cluttering the system, unresolved package dependencies that I felt a lot many times that I am spending too much time maintaining my Linux system than actually doing some work and had a strong desire to switch back to Windows.

2. Flexibility with the choice of installation of debian itself

Debian offers various ways to install Debian. One can choose either installing directly from internet with a minimal net install CD or one can install using the CD or DVD downloaded from internet or even from CD / DVD images downloaded using Jigdo. I think so is the case with other distributions, but I found installation of Debian offers much more flexibility. There are distributions like Gentoo which even build complete packages from sources but I believe that it is a waste of time and bandwidth unless you really want every bit of your computer resource to be utilized to its maximum possible extent.

3. Very active community of Users and Developers

The Debian users / developers community is so huge and active that most problems that you are going to face while installation, would have already been resolved or someone from the community will be able to help you out.

4. Larger support base

A lot of companies which provide support for their H/W on Linux, usually provide support for Debian apart from other major Linux distributions. That is at least they provide Debian specific packages or installers which specifically support Debian.

5. More suitable for programming and development

In the past having used SuSE, which offered a host of gizmo stuff for managing, installation and other utilities by default, it made me feel that I was no longer in control of my Linux System. Everything was so opaque and hidden from the user. Debian is simple in comparison and which is why it is so powerful. I really feel I am in control of my system.

6. Ports available for a large number of architectures and target device

Debian is available for a variety of architectures and platforms, which further increases the user base as well as the support base. Ports of debian exist for most common architectures like ARM, MIPS, PPC, x86, x86_64 etc. There are variety of GNU/Linux distribution which are Debian based, most notable of them is Ubuntoo, which further increases the support and user base.

The net result of having so many ports of debian is that one can run Debian on systems ranging from Servers to Desktops to Embedded Devices.

Some Debian specific useful links

http://www.debian-administration.org/

http://wiki.debian.org/

http://www.emdebian.org/

http://atterer.net/jigdo/

Android Overview

Why Android?

From what it seems to me, Google had launched this platform not because Google wants to make any profit by directly selling the platform but the key idea is to have the reach of google search beyond desktops to mobiles, PDAs and other similar devices. The benefit that it incurs to Google is pretty obvious, that is, Google Ads to each and every device using Android platform.

What Android Isn’t?

  • It is not really another GNU/Linux distribution.
  • It is not a desktop Operating System.
  • It is not the holy grail to solve all your problems.

What Android is?

Android is a Linux Kernel based OS primarily targeted for small and memory limited devices.

Why Another OS specially Linux based when there are plenty of proprietary OS and flavors of embedded linux?

The trouble with proprietary Operating Systems / platforms:

The trouble with proprietary OS is that they are closed source and access to them is limited:). Until now if you really had to write an application for a device running a proprietary OS, then it would require you to have a hardware device + license to proprietary SDK which would provide tools for building, testing and packaging the apps and sometimes you would need to pay a royalty fees for your application to the OS/ Hardware vendor. Though this situation is changing and there are couple of platforms like Maemo, Moblin, Mobilinux, OpenMoko, BadaOS, most of which are based on open source software but currently there isn’t much thrust that is behind any of these projects.

Trouble with Open Source Operating Systems/ Platforms

As such there is no trouble with Embedded Linux based distributions but for an application developer, there isn’t a consistency that exists on Embedded Linux platforms. From the perspective of application developer, one could not be sure that a certain device, library or feature would be definitely present on the device for which he wants to write the application.

Trouble with Embedded platforms (Open or Closed)

There was no way an application developer could write an application for a particular device, without getting directly involved with the device / OS manufacturer. If someone had the idea of a device or an application for small consumer device, then to build such a device required buying very expensive embedded platforms, lot of software support, cross compilation toolchains, OS, customized bootloaders,  customized kernel, Board support packages etc. Though some of the software components would come for free with the hardware, but the cost of the hardware was so enormous that it was virtually impossible for an individual to even make a prototype of an embedded device. The second problem was that there was no ecosystem in place until now which one could use to develop applications for embedded devices. Thus most device manufacturing companies ended up making complete software platforms for the device with required hardware, software features. There was no contribution or involvement from community in this cycle. This trend really got broken with Iphone where Apple opened up the application development platform to developers. Anyone who could own a Mac book and an Iphone, could now write an application and sell it in the market. Soon after the success of this model, now most platforms like Maemo, BadaOS etc. are trying to follow almost the same model.

The idea behind, Google acquiring Android Inc., the company which made Android, was to have a unified, open platform for mobiles phones, PDA, Nettops, MIDs etc. just like what IPhone offers but at the same time making it an Open Platform for developers and OEMs/ ODMs alike.

What Android offers?

Android Core OS:

  • A modified Linux kernel
  • Libraries for graphics, multimedia, data storage, font etc.
  • Android Runtime consisting of Android Core libraries and a Java Virtual Machine
  • An Application framework for rapid development; consistent look and feel, interaction and behavior of applications.
  • Some basic applications

Android SDK and SDK tools:

  • A comprehensive set of Java APIs for application development
  • Packaging tool
  • Java code Debugging Tools
  • A QEMU based Emulator for running Android applications
  • AVD Manager, an application to manage and create Android Virtual devices
  • ADB, aka Android debugger for connecting and debugging Android Device.

Android NDK:

  • Cross Compiler and other related tools for ARMv5
  • Headers, includes and libraries for building native libraries
  • A build system to easily build native libraries for Android platform

Android Plugin:

  • An Eclipse Plugin for rapid development of Android Apps in Eclipse

Development Model

Java as default programming language

The default or the recommended way of writing apps on Android is using Java with Android provided Java APIs for most tasks. This model has probably been chosen so that there is a consistent way of writing, testing, debugging and deploying an Android application, along with the added benefit of having a consistent look and feel. There is no way one can simply port and install a C application with Main entry point to be run on the Android platform. Even for the apps that written in Java there is no main entry point.

Activity Based instead of Command Based Human <—> Device interaction

Android has abstracted out the way a User interacts with the Operating System in form of activities instead of making User -> Device interaction as file / document based. What that really means is that usually a User wants to do something with the device like making a phone call, editing a spreadsheet, adding contacts, etc. Android calls this as Activity and a user transitions from Activity to Activity.

Summary

Even with so many choices of embedded OS, the application development for a developer is not easy and most of the times developers end up solving lots of platform and systemic issues instead of focusing on application development. Android aims to solve this problem by providing an Open platform for device manufacturers and developers alike, with a comprehensive set of tools for application development, testing and debugging. It comes with all the good features of the Linux kernel with added open source packages which are useful and required in most devices. Access to h/w, devices and OS functionality is hidden to application developers behind the Android’s Java APIs. Overall the platform looks promising but there are two problems that concern most developers.

1. Performance issues as Android applications run on top of a Java Virtual Machine

2. Porting of existing native apps is not possible without rewriting them in Java

Conclusion

With Google’s backing, everyone seems to have an impression that Android will be ‘the’ OS for small and medium size devices. Surely with Google doing a lot of ‘right’ things, strong technical and management leadership, strong financial backing, everyone is bound to believe whatever move Google makes. Only the time and market dynamics will tell if Android is here to stay or not.

Hardware Accelerated Virtualization with QEMU and KVM

In this post I will take you through a series of steps for setting up hardware accelerated virtualization on your PC, if it is supported, along with some background on virtualization. The assumption that I make here is that you already have a Intel/ AMD based processor with virtualization support with a GNU/Linux installation (preferably Debian).

Although it may sound little tricky but believe me, it is not as difficult to set it up, as it seems, given you follow the right steps.

A Must Read for those are planning to buy new PC hardware

Even if you don’t intend to set up virtualization on your PC, it will be a good read and probably you will think about Virtualization support before buying a new PC.

Why should You care?

Usually people prefer to stick with an Operating System which they are more comfortable with and even though they may setup a dual boot machine, they hate to switch between them frequently. At the same time, once in a while you require the other OS for certain things…for example to test some software, to learn an OS (for example if you are a Windows user and want to familiarize yourself with GNU/Linux), to isolate an environment for a software (say a crack for a software which you do not want to infect your windows installation 🙂 ) etc. , Thus having Hardware Accelerated Virtualization will definitely make the experience of using a Virtual or Guest OS a breeze and will avoid any kind of frustation due to slow emualation without h/w accelerated virtualization.

What is Virtualization?

Virtualization is a mechanism which allows running one kind of environment inside another environment. It can be of several types:

1. API Virtualization:

It is quite like cygwin, where GNU/Linux API calls are translated to equivalent windows API calls. But the apps need recompilation to run on cygwin. Another example is WINE (windows emulator) which does the reverse, i.e it runs unmodified windows applications to be run on Linux.

2. OS Virtualization / Full Virtualization:

It is the kind of virtualization where a complete Operating System runs within another Operating System. It could be any combination of OS (Linux inside Linux, Windows inside Linux, Windows inside Windows with 32, b4 bit combinations as well). it is most popular form of virtualization and is usually required by a lot of people for testing softwares. Examples of this kind of virtualization are: QEMU, VMWare workstation

3. OS Level Virtualization:

It is the application level virtualization provided by the OS itself for the applications. Examples of this include ‘chroot’ jail, where an application runs inside a ‘jail’ with access to the OS resources only in that jail. This is also useful for application testing.

4. Emulators:

This kind of virtualization is usually referred, if not limited to, to the kind of virtualization where a complete environment is replicated. An example of this is Palm OS emulator, Android emulator, Nintendo Emulator etc. Even QEMU is an example of Emulator but it is usually referred to in the context of x86 instruction emulation or OS /Full Virtualization.

5. Desktop Virtualization:

In this kind of virtualization the desktop of a user is emulated / exported to another remote system from where it can be manipulated. Examples of this are: VNC viewer, Windows Remote Desktop. I would even consider eyeOS to be a kind of desktop virtualization.

For a detailed and complete list of different types of Virtualization, refer to Wikipedia article http://en.wikipedia.org/wiki/Virtualization

Hardware Accelerated Virtualization

Hardware Accelerated Virtualization is a kind of Full Virtualization which is accelerated with some hardware support. Not too long ago, it was difficult to do emulation of certain kind of instructions, without modifications to the Guest OS. The requirement for a system to be really called Full Virtualization system is that the Guest OS shouldn’t be modified. Apparently doing this was not so easy and even if implemented, the instructions were trapped and then emulated in software itself. This made the performance of a Virtual OS really slow in comparison to the native OS running on the system.

Hardware accelerated virtualization, solved this problem by proving certain instructions / extensions so that the ‘trap and emulate’ of instructions could be done in hardware. AMD and Intel processor added extensions called vmx or svm respectively to provide support for hardware assisted virtualization. To know if your processor supports these extensions type the following on a shell prompt:

$cat /proc/cpuinfo | grep “vmx \| svm”


flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc arch_perfmon pebs bts aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm sse4_1 xsave lahf_lm tpr_shadow vnmi flexpriority

flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc arch_perfmon pebs bts aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm sse4_1 xsave lahf_lm tpr_shadow vnmi flexpriority

Essentially the command above lists the info about your CPU and greps for flag which tells if virtualization extension is supported or not.

In short Hardware Accelerated Virtualization dramatically increases the speed of a ‘Guest OS’, so that it is as much responsive and usable as the Native / Host OS.

Hardware Requirements for Hardware Accelerated virtualization

The requirements for hardware accelerated virtualization are listed below:

  1. CPU support (found using cat /proc/cpuinfo | grep “vmx \| svm”)
  2. BIOS support –> Usually most recent Motherboards which support Core2 Duo processors have virtualization support but cross check this information by reading the motherboard manual (available online for most popular motherboard manufacturers).

Software Requirements for Hardware Accelerated virtualization

  1. QEMU –> Qemu is the emulator which acts as the emulator on which the virtualized or ‘Guest’ OS runs. It is similar to VMware client
  2. Linux kernel supporting kvm
  3. kvm.ko and kvm_intel.ko or kvm_amd.ko kernel modules which utilize the vmx | svm capability for h/w acceleration
  4. An OS image to run / test.

How to set up hardware virtualization:

Assuming you have hardware which is capable of accelerating virtualization,

Warning: Since the exact steps to setup hardware virtualization could vary from one Linux distribution to another, the steps below are only indicative and you may need to figure out how to set it up for your distribution. But the principle remains the same. The configuration on which I have been able to setup virtualization is listed below:

Intel Core2 Duo E8400 (3.0 Ghz). –> supports vmx extension

Asus P5P43TD motherboard –> BIOS for this motherboard does support virtualization and is enabled by default)

Debian 5.0.3 (lenny)

At the time of writing this post, kvm support in Qemu downloaded from Debian repository (0.9.1-10lenny1) is broken. Ideally one should be able to download and install Qemu and run it without any issues and Qemu will automatically use h/w acceleration, but that is not the case and hence you require some changes to your setup.

  • Compile 2.6.32 kernel, with Virtualization support enabled. Please note that the kernel that we build here is not the official Debian kernel wherein certain code is removed from mainline kernel to keep up with the Debian software policy.

      • $tar -jvxf linux-2.6.32.2.tar.bz2
      • cd linux-2.6.32.2
    • Copy your existing kernel config to untarred folder i.e. linux-2.6.32.2/.config
    • Run menuconfig, which is a ncurses based configuration chooser (lets you choose various features to be enabled / disabled in the Linux kernel). If it asks for questions for enabling / disabling of features, just choose the default, unless you know what you are doing.
      • make menuconfig
    • Select Virtualization support in the menuconfig (see the screenshots below).
    • Install any build dependencies. This will install g++
      • sudo apt-get install build-essential fakeroot
    • Compile and install the kernel the Debian way or you can compile it the usual way and install it (in that case skip this step). The following step will build the kernel and automatically do the changes to your bootloader (grub or lilo), copy the kernel modules into /lib/modules/2.6.32.2/, setup links in / pointing to appropriate vmlinuz, prepare initrd automatically. This actually makes building kernel very easy on a debian system.
      • fakeroot make-kpkg –initrd –revision=custom.1.0 kernel_image
      • sudo dpkg -i ../linux-image-2.6.18_custom.1.0_i386.deb

make menuconfig screen - 2

make menuconfig screen - 2

Screenshot when kernel source is configured with make menuconfig for virtualization support

  • Download, build and install the qemu-kvm package

    • Download the qemu-kvm package from http://sourceforge.net/projects/kvm/files/. In my case I downloaded qemu-kvm-0.12.1.1.tar.gz, which works fine for me.
    • Ensure that package checkinstall is already installed on your system. This is not essential as you can install a package on your system using usual ‘make install’ which will simply copy the required files in appropriate folders. But installing it through checkinstall ensures that you can get the status of your installed package and you can remove it like any standard debian package. To install ‘checkinstall’:
      • sudo apt-get install checkinstall
    • Do
      • $qemu-kvm-0.12.1.1.tar.gz
      • cd qemu-kvm-0.12.1.1
      • ./configure
      • make
      • checkinstall
    • Chekinstall asks some questions like the name of the package etc. which you can provide so as to keep track of the installed packages.
    • Once you do dpkg –list it shows the following:
      • dpkg –list qemu-kvm
        Desired=Unknown/Install/Remove/Purge/Hold
        | Status=Not/Inst/Cfg-files/Unpacked/Failed-cfg/Half-inst/trig-aWait/Trig-pend
        |/ Err?=(none)/Hold/Reinst-required/X=both-problems (Status,Err: uppercase=bad)
        ||/ Name                              Version                           Description
        +++-=================================-=================================-==================================================================================
        ii  qemu-kvm                          0.12.1.1-1                        qemu-self-compiled
    • To see the list of files installed by this package, run the folowing command:
      • dpkg –list qemu-kvm
      • This lists a lot of files of which the following is important
      • ….
      • …..
      • …..
      • /usr/local/bin/qemu-system-x86_64
        /usr/local/bin/qemu-nbd
        /usr/local/bin/qemu-io
        /usr/local/bin/qemu-img
        /etc/init.d/qemu-kvm
  • Now reboot your system
  • Add the user to kvm group. This is essential so that the qemu can read the device /dev/kvm, which by default has permissions 600. Simply changing the permission to 666 is not a good option as this will be reset to 600 on next reboot.
    • sudo adduser divkis01 kvm
  • Before you can us QEMU + KVM, you need to have an OS image to be able to boot it with QEMU. To prepare images,
  • First make an ISO image of your XP, Ubuntoo CD or alternatively if you already have an Iso Image skip this step
    • dd if=/dev/cdrom of=~/MyMachines/winxp.iso
  • Now create an image in which the OS installation will be done.
    • qemu-img create -f qcow2 winxp_x86.img 10G
  • Install the OS on the image created in last step
    • qemu-system-x86_64 -hda winxp_x86.img -cdrom ~/MyMachines/winxp.iso -boot d -m 512
  • Once the installation is complete, you are ready to run Qemu with hardware acceleration. You can boot the OS using the following command:
    • qemu-system-x86_64 ~/MyMachines/windows/winxp_x86.img

See the screen shots below which shows Windows XP and Ubuntoo booted on a Debian GNU/Linux host

QEMU running Windows and Ubuntoo images on Debian host

Caveats

  • If you see that your native OS (on which you are running QMU) is fast enough but the guest OS runing on QEMU is too slow, then it is possible that hardware acceleration is not being used. There are several ways to figure out if that is the case or not.
    • Listing the currently loaded module and grepping for kvm, should show non zero entries in the right most column. For example on my machine, it shows:
      • lsmod | grep kvm
        kvm_intel              31979  6
        kvm                   181149  1 kvm_intel
    • On running QEMU, if QEMU complains with the following message
      • qemu-system-x86_64 ~/MyMachines/Linux/ubuntoo-9.10.x86.img &
        divkis01@divkis01-pc:/home/data/downloads/isos/debian-x86$ open /dev/kvm: No such file or directory
        Could not initialize KVM, will disable KVM support
    • The performance is too low of the guest OS. I have found drastic difference at least 10x perceived difference with and without hardware acceleration
  • The reason for h/w acceleration not available could be due to any of the following reasons:
    • kvm not properly installed
    • user running QEMU is not part of kvm group i.e. the user does not have access to device /dev/kvm
    • kvm is installed but the kernel modules for kvm (kvm.ko and kvm_intel.ko / kvm_amd.ko) are not properly loaded
    • The installed kernel does not have kvm built as module
    • Or probably h/w acceleration is not supported by your processor or motherboard.

Useful Links

http://en.wikipedia.org/wiki/Hardware-assisted_virtualization

http://www.linux-kvm.org/page/Main_Page

http://www.nongnu.org/qemu/user-doc.html

http://kernel-handbook.alioth.debian.org/

http://wiki.debian.org/QEMU

If you face any difficulty setting up kvm or QEMU, don’t hesitate to send me questions. I will be glad to help. And if you find this page useful, please feel free to drop a comment.

Happy QEMUing!!!!

    2. Download, build and install the qemu-kvm package