Feb 182018

My first desktop was Solaris. In fact, I had two Solaris monitors on my desktop, back when I was at Bell Labs. The ostensible reason was to code. Never mind that they were black and white. Still, I could use them for coding. I was automating anything that I could automate. In fact, I wrote some code to simulate remote execution protocol on SMPT + ppp.

When Linux came, I was one of the first to start using it in my circles. I liked its familiarity, its ability, and its generosity to let me tinker. And, tinker I did, starting with dabbling in mmap code. The ability to use the same setup for my personal and professional work made be doubly productive.

Once I moved to windows, my life changed. I started separating my professional work and personal work. As I stopped writing code for my personal work, my professional work too ended up relying on professional tools. “This is the way the enterprise works”, I would tell myself. “I am too senior to muck around in code” I would tell myself.

Even if I used Linux on laptops, the Microsoft office tools forced me to use Windows. Virtualization was not good enough either; so I ended up using mainly Windows. Until I bought a Mac in 2008. Since then, I only used macbook family at home. To me, it is blend of the both worlds. I can use command line the way I want to. And, I can get all expected office productivity tools. To add bonus, the font rendering on OS X is vastly superior to windows or Linux.


Alas, my desktop still languishes in Windows world. I thought about Hackintosh, but that felt like an exercise in masochism. I thought of getting a mac for desktop, but I already have a setup with 64 GB, 8 core machine and dual monitor setup. I use virtual machines, but the inertia and convenience pulls me into Windows.

Well, no more!

Over the weekend, I went back to Linux. It has been a while for me to use it as a desktop, but I am happier after the move. I tried Arch Linux, first. I thought it would be a better fit for my hacking tastes. After spending a day with it, it felt like too much of wild west. Based on prior experience, I chose to go with Linux mint. Once I installed the graphic drivers, printer drivers, the desktop was functionally equivalent to the one I had under Linux.

Now, here is my setup, which I did, say 4 years back:

  1. 2x 27 Inch desktop with 2560×1440 resolution
  2. 500 GB ssd, with external Hard drive of 3 TB
  3. 64GB RAM
  4. NVIDIA Geoforce GT 640 as the graphics card
  5. i7 3830 CPU (old one) – 8 threads

Here is the way I am working now:


  1. The Linux machine is my workhorse. I have my ssh keys installed so that I can access any of my machines from here. Especially, my final work happens on my GPU server.
  2. I have VMPlayer installed. I set up my Windows 10 so that I can do my non-development work:
    1. MS Office installed so that I can handle the usual work related mails.
    2. Xmind for my other documentation work.
    3. Dropbox so that I can access my files from anywhere.
    4. Onedrive – so that I can access other materials.
    5. Livewriter for writing blog posts.
  3. On my Linux box, this is what I use:
    1. Docker setup
      1. I use portrainer so that I can manage my container.
      2. I have docker containers for most of my work needs.
    2. Pandoc for my regular documentation needs – I use markdown for most of my documentation.
    3. PyCharm for any python development. With proper configuration, I can check for syntax errors before I run the code.
    4. Visual studio Code for any other editing needs. I switched over from Atom, because VS code seems to have better developer support.
    5. File sharing between my Linux and Windows with SMB. Unfortunately, dropbox does not work with networked file system. So, I have to keep the primary on Windows.
    6. Anacoda 5.x so that I can do my python development.
  4. To tweak:
    1. I want to tweak my fonts setup so that the reading experience is as good as OS X.
    2. I want to tweak my rsync setup so that I can use remote sync for my desktop
    3. Organize my work documentation: Choices – to use Google docs or gitlab. Not sure what to use.

Now, I find myself coding and documenting the way I used to. Linux brings me back my youthful days!

 Posted by at 6:59 pm
Sep 272017

When I was a student at IIT Madras, my dream place to work was Bell Labs. Eventually, after a PhD, I ended up there. One of the preliminaries was getting a phone instrument. They told me that it would take a week to get a phone assigned. A world class phone company, taking that long to get a phone? If it was left to me, I reasoned, I would get the phone under a day.

When I co-founded Savera, a telecom interconnect billing company, I had the chance to put my theory to action. I was in-charge our IT and phone systems. I had fun getting an UnPBX. I also managed to get the phone ordering under a day. After a few employees, I let others handle the ordering. Soon, we ended up with many different models of phones. Sometimes we got the phone under a day and sometimes it would take many days. It was frustrating for our HR — they could not plan the on-boarding process. Eventually, I relented and had our professional IT manager, who setup a process for procurement of phones. Even if the procurement time increased to a week, it was always guaranteed. I learnt a valuable lesson.

Now, to standardization.

We are sent conflicting signals throughout schooling. On one hand, we are supposed to follow rules: school rules, arithmetic rules, L’Hospital’s Rule, whatever. On the other hand, we are constantly exhorted to think outside the box. We are asked to break out of the crowd. Only in the later years, we see how this dichotomy shapes our world. Unless we manage both sides — standardization and innovation, we will not succeed, as a species.

Case for standardization

We all understand the need for standardization. Despite what all the fiction books or movies tells you (“Be yourself”, “Work out your own way” …), standardization is a good thing. It is the difference between a proof of concept to scaling to millions. It is the difference between a startup and an enterprise. It is the difference between a small group of brilliant people and a large group of people working as a team.waiting

Standardization helps in many ways:

  1. It leads to predictable outcomes: Standardization aims for measurable, clear outcomes. For instance, if we standardized the process of getting a new phone, we can assign responsibilities, SLAs (say, how long to do what and what number of mistakes allowed), then we know what to expect and when to expect. That increases customer satisfaction (Imagine being told that the cable person comes between 10am-2pm and you need to waste a whole day waiting for them. Or, if they say 3:45 pm, even if it is late, you feel more empowered.)
    1. It lets us plan better: The predictability of outcomes lets us plan better. We can set the expectations for the customers. We can plan the next steps and dependent tasks better.
    2. It lets us build a large supporting system: If we have predictability, we will not mind spending effort in building a support system. This is how ecosystems are built.
    3. It lets us create complex systems: People who build complex systems understand that variance in a system increases with the variance in constituent parts. If there are 10 steps, and each step could take +/- one week, on the whole, the final system ends up taking +/- 10 weeks. That degree of variance makes the system unviable.
  2. It reduces costs: The cost reduction shows up for many reasons in many ways.
    1. It reduces variations: Instead of several options and choices, by standardizing on a few, it reduces the costs.
    2. It lets people spend least amount of energy for reasonable outcomes: Standardization allows us to create standard operating processes, tools, and templates to reduce the costs.

assembly-lineLet us look at different examples, including information technology on how standardization helped progress.

  1. Growth of cities: The growth of cities became possible with standardization of different services. In fact, these shared standard services, from security to sewer cleaning, from transportation to trash removal service, from piano tuning to patio furniture let people specialize and scale up the services for large masses. All of these services made for the megacities of the day.
  2. Standard money delivery: In the medieval ages, sending news or money from place to place was difficult. The Templar order made it possible to send money to different places: Say you need send ransom for your uncle’s release in Turkey. You give money to the local Templars in Birmingham and the agent in Turkey pays locally. The guarantees and the timelines they offer lets people plan the rest of the actions. (I am thinking . In fact, one argument against bribe is not the cost, but the unpredictability of service.
  3. boxStandard containers: In the book Box, the author shows how standardization lead to the support systems like mechanical loading and unloading. All in all, this standardization pretty much created the current global supply chains and the globalization. A shirt can be made in china and button sewn in Thailand, and label put in Japan. All of this is possible only with standardization.
  4. GUI Programming: I remember the days of early 90s, when GUI programming in X-Windows and Motif were very difficult. Once people figured out the standard way to build applications, tools like TK and VB drove the price down. The myriad choices of Motif became standardized with fewer ways in the modern toolkits. That led to easier paradigms of programming making GUI programming within the reach of mere novices.

All in all, standardization plays in many ways in spreading innovation to the masses. It is how industrialization succeeded, by creating a large class of working class as well consumers. It has a direct bearing on the success of software industry. It created the entire software development outsourcing, where offshore development companies came with standard tools, processes, and methodologies to deliver software with expected quality, cheap price, and predictable timelines.

Case against standardization

Yet, standardization hinders us in many ways. In fact, its success is the cause for its failure. By creating a standard way to work, it creates a tradition that robs incentive for innovation. This problem shows up in many ways:


  1. Standardization is resistant to change and innovation: There are many reasons why standards are resistant to change:
    1. The sunk costs: They are already here and everybody already are using the standards. For instance, moving from qwerty to alphabetic keyboard does not happen because of this inertia. Think of all the keyboards and the training costs.
    2. The risk factor: They are cheap enough that any change, even if was cheaper, carries a risk. The risk of change deters most organizations to experiment. For instance, in the early days, moving to cloud was considered risky — even if it was cheaper.
    3. The interdependencies: The collective interdependencies make change even more difficult. For instance, the size of the shipping container depends on the width of the train, which itself depends on the width of the tracks; and of course, the established trains and tracks. Nobody wants to be the first one to change.
  2. Standardization removes choices: Ultimately, in the name of standardization, we end up serving common denominator — of tastes, of choices, and of creativity. Remember the old saying: “You can have any color car, as long it is black”? It leads to monoculture of technology, culture, and aesthetics that are susceptible to viruses, exploitation, and systemic biases. For example,  the standardization on windows in front offices led to the current spate of rampant viruses and other malicious threats.
  3. Standardization is difficult to improve: Any incremental improvements to standards need to support the existing setup. That restricts the kind of improvements we can make. Besides, the enemy of the best is not worst, but “good enough”. Standards are good enough. For example, consider the USB-A port. It is not reversible; and it ends up costing humanity untold number of man-hours.
  4. Standardization naturally degrades over time: Even though standards work reasonably well, they degrade over time. Specifically, it happens with training of people. Instead of the spirit of the standards, people follow them to the letter. An orthodoxy encourages blind application of standards, which ignores changing realities and people preferences. For example, as the windows GUI programming became popular, as more and more people entered the field, the quality of GUI design went down.


[Consider the fireplace in a house. Nobody uses fireplaces anymore. Yet, almost every house has a fireplace. Instead of its functional use, it became a part of the cultural icons and a part of aesthetics. Ironically, it ended up with a different utility value, as a place for people to gather and an informal place to sit. We are constrained to keep the original designs and innovate around it.]

Overall, we see the trend: Standardization does not play well with innovation or change. It has lot of implications to the software industry, especially software services companies. The very attributes that made them a success are going work against them in the coming years.

Why? Let us see:

  1. The change in the ecosystem: The new software is delivered on new channels. They use new technologies. They use cloud. They are built on newer platforms.
  2. The change in the expectations: The new software is built for more digitally savvy users. They are expected to be user centric, with high degree of reliability and performance.
  3. The change in the requirements: The new software changes the requirements often. In fact, unlike before, the requirements come from actual users. Moreover, applications average life is getting shorter and shorter.

While software service companies are grappling with the issues of irrelevancy, the enterprises are facing the same challenges in many ways.

The dilemma for the enterprises

Let us take a typical enterprise: they started with IT department with pay roll in the 70s. The brought in PC’s for client/server programs to create small apps for their customer relationship management. They tried to write lot of apps, but eventually wisely moved to SAP. In the 2000’s, they rewrote parts of SAP in Java on the web. Later in 2010’s, they used mobile and modern web.

All in all, their focus was to take the requirements and use the right COTS application. The only changes were the integration challenges with web and the ERP; and the new channels of mobile and the web in the modern times. The focus was always to reduce the costs and stick to the service levels promised.

For them, standardization delivered. They could reduce costs; They could use average programmers; They could use standard methodologies.


Suppose you are the CIO now. You understand you cannot ride on the cost cutting alone. You cannot wait for the business to send in the full requirements. Sometimes, you work with the business; sometimes, you lead them to new possibilities. You understand that may mean lot of custom applications. How are you going to deliver those without blowing the budget?

Let us be even more specific. Suppose you are planning on delivering lot of front-end applications. Lot of them are going to be short-lived. If you are going to develop that many, you know standardization delivers the right kind of efficiency.

The problem is that these technologies are moving really fast. If you standardize and build all the ecosystem — the assets, the processes, the training, the hiring — then, what happens if the technology becomes outdated? Within JavaScript, on average, there is one new framework every quarter. Even if you choose the right one, it will be outdated soon. You don’t choose — you end up with a dozen. You choose, you still end up with a dozen, but all the ones are outside your governance.


The challenge of standardizing in fast moving world is a problem that most CIO’s, CTO’s are facing now. There are no easy answers. Still, there are lot of best practices, checklists, and ways of working that can help in balancing this standardization that can support innovation.  We will look in-depth about those practices in the next article. Specifically, we will look at:

  1. checklist-911840_960_720It is clear that standardization offers lot of benefits. Where does it work best? How do we determine where to standardize? Does it depend on the organization?
  2. How do we support innovation through standardization, if they are at logger-heads?
  3. How do we standardize in the fast-paced technology fields?
  4. What kind of standardization should we do? Is there a checklist?
  5. What are the processes associated with standardization? How do we deliver such standardization?

We will look at all those questions in the next article. Feel free to comment with your experiences. Do you relate to this challenge at all?

As an aside, I find this dichotomy play in several different arenas — notably conservative vs. progressive thought. Or, established companies and startups. Yet, I also see a convergence that balances these two aspects. I think that ought to be the goal of any modern enterprise.

[Original at: www.kanneganti.com]

Sep 162017

During the heydays of Linux, Neal Stephenson wrote an essay called “In the beginning, there was command line”, about operating systems. The point was not so much about command line; and he eventually said that he does not stand by that essay, after using OS/X. Still, after many years of using Unix, I resonate with that essay. Even if I use Mac OS/X, I find myself using command line often, for all my needs.

These days command line is becoming fashionable again. Thanks to cloud, and thanks to cloud running mostly on Linux, we have a need for command line tools. Most of the automation and orchestration does not need user interface. If any, they are doing away with user interfaces. Even when they need user interface, it is easy to throw one together with REST APIs and http.

First, a little bit of motivation for those that grew up using Windows. Yes, windows make it easy to use a system, with help and hints along with invocation of a system. However, as you are pulling together different systems, filling in the gaps between systems, you find command line will help you become more productive. You are not limited by the tools vendors offer you; you are only limited by your imagination and ability.

Particularly if you are working in cloud, you find yourself using command line often. You are logged into a remote machine – and you need to download or install new app. You need to edit files, transmit files, and change parameters. You need to do all this, possibly from even your cell phone. It can get very frustrating to look for applications to suit your workflows. And, the network settings may not even help you to run arbitrary programs – often they allow only ssh.

If you are going to develop or test applications on the cloud, you have to learn how to use command line effectively. This article is a distillation of the tools that I found useful in working on remote machines. As usual, the following caveats apply: This is not meant as a guide, but a curated list of tools that I found useful. I observed myself at work for a few days, and noted down all the tools I used – naturally, your criteria might vary.  I will only explain what the tool is and how to use it. I will leave the details to a link. Hopefully it is useful for you to become productive.

Logging in: Client

Most Linux systems on the cloud allow logging in only via ssh. If you are in OSX or Linux machine, you will not need to look for a client. On OSX, I would recommend you use iTerm2 instead of usual terminal application. On windows, you have two choices: puTTy and MobaXTerm (home edition). You can use other systems such as cygwin, but they are too heavy for our needs. All we need is a window into the remote machine.


I prefer MobaXTerm for the following reasons:

  1. Integrated sftp: Copying files back and forth from your desktop and the server is easy.
  2. Credential management: It remembers passwords, if you allow it, for easy login. A better method is to use your PKI infrastructure.
  3. Support for X: If you like, you can even tunnel Xwindow programs. Not that I recommend, because you should be able to live with just command line for your needs.
  4. Additional tools: Even if you are using it as a local terminal, it has enough goodies, almost like light weight cygwin.

Here is the way I work:  I start mobaxterm. I do ssh into the box of my choice. If I need to cut and paste a small amount of text, I would use the windows cut-n-paste mechanism. If I need to send large amount of text or data, or even downloadable files, I would use the built-in sftp mechanism to send the files back and forth.

Logging in: ssh

While mobaxterm can remember the password and enter on your behalf, it is not enough for you. Ssh became a de facto mechanism for lot of communications. For instance, if you are using git, you use ssh to login. If you are using scp, it uses ssh underneath. If you want to automate such commands, you cannot expose the password to the scripts. Instead, you would use password-less ssh, with public key/private key infrastructure. Also, you may have to login to multiple servers from your cloud gateway machine where you logged into.

All you need to do is:

  1. Generate ssh keys using ssh-keygen. You can do it anywhere and transfer the files. I usually work on many machines. I have my own generated public and private keys that I take it from machine to machine.
  2. Copy the id_rsa.pub to the remote machine into the authorized_keys in .ssh folder.
  3. When you login first time, you need to accept the machine id. After that, you can login without the password.

Debug hints: Try ssh –v or ssh –vvv to get an understanding of what is going on. If you are running into troubles, most likely it is because of the permissions issues. Make sure id_rsa file is readable only by you. And, id_rsa.pub is readable by others.


Additional hint: If you are inactive, the server logs you out after a period of time. If you like, you can configure your ssh to send a packet periodically to keep the connection alive. For that add the following lines in .ssh/config:

Host *
	ServerAliveInterval 30

Interactivity: Tmux

There are several challenges in working remotely on ssh using a window. What if you want to temporarily run a command as a root? You need to open another window. What if you want to go home and work on the remote machine? Your connection will be lost and you need to re-login again from home. And, all your context is lost.

Tmux (and an older alternative, screen) helps you with these kind of problems. There are nice user manuals, guides, cheat sheets for you to learn from. I will tell you how I use it:

  1. As soon as I login, I start a tmux session. If I already had one, I will attach to that session.
  2. In a session, I have at least four login sessions running.
    1. One is for working with command line as a user.
    2. Two is running as root – useful for installation or killing the processes.
    3. Three is to watch for essential processes
    4. Four is my editor to edit the files locally.
  3. I use C-b 0/1/2/3 to go to the window of my choice to work on it and come back.
  4. When I am done, I detach from the session.
  5. When I use a different machine, I just attach to this session.
  6. Sometimes, I run it from a shared account – and then, two of us can login to the same session and see debug any code we run.


Just a shout out for watch: Suppose you are planning on running a command in a loop and watch its output. Watch runs it in a terminal and repaints only the changed characters. That clearly shows what part of output changed and what did not. For instance, you can run “watch –n 1 ls –l” runs the command “ls –l” every second. You will see clearly if there is a new file that appeared. It is more useful for checking the status, or the load of the system. Very useful for checking the load on my GPU machine, using “watch –n 1 nvidia-smi”.

Downloading: wget, curl, elinks

The first thing you will need often is to install a package. These two programs help download the any software or application that you want. Out of these, I prefer wget.

Lot of times, you may have to login or accept a EULA before downloading the software. In that case, you have two choices: Download the software onto your machine and use scp to copy the file. Of course, if your network may not be good enough to download.  Or, use text based browser like elinks to download the software. Of course, modern JS based websites do not work well with elinks. In that case, if you used MobaXTerm, you may be able to run X-windows application such as firefox and have it pop up on your terminal.

Rsync: Swiss army knife for moving files between machine

Let me give the scenario. Suppose you are editing files on your local machine. You want to move them to a remote machine. How? You can use scp or sftp. Suppose you changed a few of the files in a folder. How do you move them? It is complicated to remember the path and move to the right path. Or, if you changed one line in a large file, you still have to copy large file.

The answer to this and many other questions is rsync. It can work over ssh, which means, if you can login, you can move the files. It moves only diffs, which means you get to move only the differences.

I have a system to publish a website. I edit the files locally. I wrote shell scripts that rsyncs to remote location and that shows up in the website.  For instance consider the following one liner:

rsync -avz -e ssh --delete ./local/ root@machine.com:$remote

It uses ssh (-e option), –z (compression), –v (verbose), and –a (archive mode). The “-a” does the trick. It syncs 1) recursively, 2) copies symlinks, 3) preserves owner, 4) preserve modification times, and 5) preserves permissions. What it means is when it copies the folders, it does the right thing. The –delete option removes extraneous files from remote location. That is, if you delete a file local directory, it will remove it from remote too. Without it, it only updates the files. The trailing slack in the local folder means, copy only the contents. Without the trailing slash, it will copy the folder, including the contents. While it looks magical, it only touches the files that modified, and only transmits the deltas, and also compresses.

Example use case: What if we want to update the remote folder every time I write to a local folder?

What we need is to monitor a file or folder for change. There are several ways we can do it.

  • You can use inotify tools for any change monitoring. The link provided gives you an idea on how to use it for continuous backups.
  • My favorite is to use node.js – there is a module called chokidar that does what we need and more. You can invoke any process on any change.
  • Python has a module called watchdog, which does something similar.

All we need to do is to use any of these programs and keep in a loop for changes. When change occurs, do the remote push and then go back to the loop. The rest is left as an exercise.

Find: Swiss army knife to work on many files

For the life of me, I can never remember the syntax for find. Still, I find myself using this utility all the time. It is useful tool while dealing with lot of files at one go. Let me illustrate with some examples:

# To print all the files
find .

# To print all the files with .js extension
find . | grep .js$

# To print only the directory files
find . -type d

# To print only the non-directory files
find . -type f

# To print all the files with a specific string, say "TODO:"
find . -type f | xargs grep "TODO"

# Notice that xargs repeats the function on all the arguments.

# To print all the files that changed in the last 2 days
find . -mtime 2

# You can combine all the capabilities in many ways
# For example, all the java files touched in the last week,
# with a case insensitive TODO
find . -type f -mtime 7 | xargs grep -i TODO

Admittedly, there are better ways to use ‘find’, but I do not want to remember its idiosyncratic options – these ones are the most useful, and most common in my experience.

In addition to the find, there are other unix utils that I find myself using a lot. If you know these, you can use the command line effectively, only reaching for higher scripting languages when really needed. Simple bash, with these utilities can do all that you want and more. Moreover, all these are available in any standard Linux installation. For instance, using cut, paste, join, you can selectively choose and merge parts of the file. You can sort files based on numbers. You can slide and dice files with it.

Running a website: Python, live-server

You have a static website that you want to show to a colleague. Or, you have a set of files that you want to serve up for an application or for colleagues. You don’t have to go all the way, installing and configuring Apache. Instead, the following one-liner can do it:

# python2

python -m SimpleHTTPServer

# python3

python3 -m http.server

# Standard port is 8000. 
# You can change it by specifying the port as an argument.

If you do not have Python installed, you could try node.js based http servers. The two servers that I like are: http-server and live-server. The beauty of live-server is that the browser auto loads if the file it is showing changes.

# To install and use http-server
# Default port is 8080
npm install -g http-server

# To install and use live-server
npm install -g live-server

Live-server is particularly useful when we develop static websites. As we edit it, and push it to the remote site, the browser automatically get updated.

Running a website from behind a firewall

Sometimes, your machine may not be accessible from outside. It may be behind a firewall – you may have had to go through a series of ssh tunnels to get to that machine. Or, your machine may be in the corporate VPN.

Fear not. As long as you can access a public website, you can show your website to the outer world. You have two options: http://localtunnel.me – you need to have nodejs installed on your machine.

# Install localtunnel
npm install -g localtunnel

# Start the webserver
live-server --port=8000

# Run the command to tunnel through
lt --port 8000

# The system will give a URL for you to access it from outside.
# Example: https://nuwjsywevx.localtunnel.me/
# Notice that it is publcly resolved URL
# And, it comes with https!

You have other options: You can use http://ngrok.com (There is free plan). You can tinker with your ssh to do reverse tunneling. I am going to give the full details here, but if you use “shell in a box”, you can even access this machine from outside the firewall, via secure shell. Naturally, please comply with corporate security guidelines.

Creating and publishing content: markdown tools

For programmers, creating documentation is important. They need to record what they do. They need to write design documentation. They need to write user manuals.

Writing in word or any other WYSIWYG tool is pointless – it does not work well over the command line. Suppose you are on a remote machine – and, you need to refer to installation manual. If it is in .docx format, how are you supposed to read it from the command line?

Of course, you could write your documentation in text file. It is ubiquitous; you can create and read it from many tools. Yet, you will not get any good looking and useful documentation out of it. You cannot print the documentation. You cannot have images. You cannot use hyperlinks. All in all, your experience is only limited to the command line. It cannot be repurposed.

A midway is markdown. It is such a simple format (built on text) that it is easily readable, even if you do not do any conversions. It can easily be created on any tool (like any text editor, though I use atom or Visual studio code. Of course, you can use even notepad.

The rules are simple: See https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet  — you can learn them in 10 minutes. If you use atom or code, you can even see preview as you edit.

Once you author the document in markdown format, you can convert to any format you wish. There are several tools. My choice tool is pandoc. It can convert markdown into many formats, including docx (for word), or text, or html.


You can do even more with markdown. I use markdown with minimal yaml settings to generate websites – I use mkdocs for its simplicity.  If you have a bunch of markdown files, you can generate a static website, with themes and searching capability, using mkdocs and host it, using the technics we discussed earlier. It is always handy for a developer to create a simple website on that favorite library or design that they are working on.

Working together: Expect, tmux

Suppose you are a remote machine – and you are experiencing some trouble – some command line application is not working. You want to show it to an experienced colleague. Or, you just want to train a junior colleague. Here are the two ways:

  1. Suppose you are sharing the account (it is common with testers etc to create a common account and use it). In that case, you start a tmux session like we described earlier. Let the other colleague also login to the same session. Now you can easily see what the other is seeing. Of course, you want to keep the terminal size the same. Otherwise, it will look weird.
  2. If you have a different account, you still can share your session with others. What you use is a program called “kibitz”, which is a part of “expect” system. Assume you want help from user <abc>. Both of you should be logged in at the same time. All you need to do is kibitz <abc> and then <abc> needs to type the command that shows up on their terminal. From then on, they can see and type on your terminal along with you.


I hope you will find these commands useful. Using this knowledge, you can edit files locally, push them to remote machine, and share them on the web with a client.  You can start working at the office and continue to work at home on the same setup. You can keep machines in sync. You can create developer friendly websites and share it with colleagues from behind the firewalls.

Finally, there is saying: In windows, you are limited by the tools you have; in Unix, you are limited by the knowledge you have. All the tools are there for you to pick and use, combining in many different ways. The more time you spend, the more proficient you become. Go forth and learn!

 Posted by at 10:41 pm  Tagged with:
Aug 182017

As I have been dabbling in machine learning, I decided to build a GPU machine. Over the weekend, I built one for myself. I wanted to share the details with others so that they can make the right choices.

A key part of machine learning is to do lot of mathematical computations. As you are training the neural network, you need to do large number of matrix multiplications. In fact, almost all the computations are on matrices — these transformations are similar in nature to graphics. Say you want to tilt an image? Lighten or darken an image? All of it is matrix operations. Graphics cards (Graphics Processing Unit of GPU) excel in these computations.
So, I wanted to build a high end GPU machine for as little money as possible.

Getting the parts

The challenge with getting the parts is how do you know which parts work together? How do you know what all you need? Thankfully, www.pcpartpicker.com will come to the rescue (there are some limitations, which I will describe later). If you are impatient, here is my list:
PCPartPicker part list / Price breakdown by merchant

Type Item Price
CPU Intel – Core i5-7600K 3.8GHz Quad-Core Processor $211.98 @ Amazon
CPU Cooler Cooler Master – Hyper 212 EVO 82.9 CFM Sleeve Bearing CPU Cooler $24.88 @ OutletPC
Motherboard Gigabyte – GA-Z270X-Ultra Gaming ATX LGA1151 Motherboard $158.88 @ OutletPC
Memory G.Skill – Ripjaws V Series 64GB (4 x 16GB) DDR4-2400 Memory $511.99 @ Newegg
Storage Samsung – 850 EVO-Series 500GB 2.5″ Solid State Drive $174.99 @ Newegg
Video Card EVGA – GeForce GTX 1080 Ti 11GB SC Black Edition Video Card $724.99 @ SuperBiiz
Case Corsair – 330R Quiet ATX Mid Tower Case $49.00
Power Supply Rosewill – 600W 80+ Bronze Certified Semi-Modular ATX Power Supply $58.87 @ OutletPC
Prices include shipping, taxes, rebates, and discounts
Total (before mail-in rebates) $1935.58
Mail-in rebates -$20.00
Total $1915.58
Generated by PCPartPicker 2017-08-18 22:19 EDT-0400

https://pcpartpicker.com/list/GpFnD8 [Knowing what I know, this list would work — but I would get low profile memory, as it was difficult to fix the CPU fan with the these high memory sticks].


First, what is the right build for GPU computing? The first and foremost is that you need a good GPU. As of writing this note, it is Nvidia 1080 Ti. Here are the reasons why:

  1. Nvidia is the undisputed leader in GPU computing. While AMD is a good brand, the library support for Nvidia is unmatched. It is well-debugged, with full support for many frameworks.
  2. Now which card? Within the home buyer’s range, you are looking at GeForce GTX series. The higher the number the better it is. Ti, standing for Titanium offer upto 30% more than the regular version.  You can get 970, 980, 980 Ti, 1070, 1080, 1080 Ti — any of them will work. Realistically, these options stand out:
    1. 1080 Ti: This is the best one currently. Runs around $700
    2. 1080: 30% less than the Ti. Runs around $500
    3. 980: You can get them on Ebay — if you are lucky, you get it under $150.

I think the best value is for $1080 Ti, if you are a serious player in machine learning. For full details, please visit: http://timdettmers.com/2017/04/09/which-gpu-for-deep-learning/


The GPU GeForce 1080 Ti is available from many vendors like EVGA, MSI, ASUS etc. The main differences are all the brands is:

  1. Ability to overclock
  2. Number of fans: Typically most have two fans. Some have 3 fans allowing for higher clock speed. Some even have liquid cooling for quiet operation.
  3. Connections: Display ports, HDMI or DVI-D’s that these motherboards can connect to.
  4. Color schemes (don’t laugh — lot of gamers expose the innards of the machine)

I ended up with EVGA GeForce 1080 Ti SC Gaming edition. There are higher brands like SC2 — but, they did not offer much more.

CPU + Motherboard

While AMD Ryzen is really attractive now, I opted for Intel. Intel works without any tweaks with all the software we use, including Linux and virtualization. I will use AMD Ryzen someday, but not yet. AMD Ryzen 7 on the higher end can offer 8 cores or 16 threads. It offers great performance, yet the stock Linux does not work with it (the latest kernel does work).

You need to know a few things before buying a CPU. CPUs come in two varieties: the one that can be overclocked and one that can’t be. In the case of Intel, it is Intel K series can be overclocked. In Ryzen it is X series. The current generation of Intel is Kaby Lake (7th generation). The older one is sky lake (6th generation). While there is no major performance difference, I did not find much cost difference either.


The overclocked versions have the advantage of supporting overclocking. I am not sure if needed it. Moreover, the standard version comes with a CPU fan and overclocked version needs a separate CPU fan.

Still my micro center was selling i5-7600K for really cheap. I went with it.

The mother board needs to support the Kaby Lake (7th generation). It also needs to support the GPU length. Other than these two, most are standard. Gigabyte has good reputation of motherboards. I chose this one because it is on sale. Notice the SLI — that means, it can take two GPU’s, if I want to up the performance later on.


Some motherboards support m2 SSD’s, which are much faster than usual SSD’s. They fit like memory, and act like disks. Still, they are expensive and not yet mainstream. This motherboards supports m2 SSD’s if I need to use in the future.


The other big factor in building a computer is the case. Since you chose ATX motherboard, you are looking for a mid or full case. You also want it to be quiet. You want it to have at least a few fans (like two or three). You want it to have enough space to organize the cables, fit the disks etc. In my case, all I needed was just the SSD hard disk. My priority was a quiet box. I went with Corsair 330R silent case. It has padding on the sides and two fans (with support to add more fans, if needed).


This case does not come with a PSU. I looked for the integrated cases, but did not find any good ones.


The choice of PSU is largely boils down to these:

  1. Modular vs. Semi modular vs. non modular: The PSU comes with many power cables. Non-modular PSU has all the cables attached, whether you use them or not. Modular has all the cables as optional. If you don’t use it, you don’t need to clutter the case with attachments. I prefer the semi-modular, with the essential ones already fixed (we do have to power the CPU and motherboard, right?)
  2. The certification: Any kind of certification here is good. The lowest level 80+ Bronze is good enough for us. This filter weeds out fly-by-night operators.
  3. The wattage: The PC partpicker will tell you how much power source you need. For the GPU we chose, the recommended wattage is 600W.


I ended up picking up one from local microcenter that was on sale that meets these requirements. The red one needs to be used for the GPU. Two means, it supports two GPUs. One of them is SATA connector for connecting the disks and optical drive.


These motherboards can take up to 64 GB. In my experience that is ample, even if you are running many processes. In reality, we will be limited by GPU anyways. This motherboard takes 128pin DIMMS, with speeds from 2400 to 3600. There isn’t much difference in performance, according to the research I did. The difference may be 1% or 2% — which is not justified in the increase in the cost of 30% of the memory. So, I ended up getting the DDR4 2400. Look for any deals on usual sites.


One caveat about memory. The memory goes next to the CPU on the motherboard. It only takes 4 cards — so, you should get 4x16GB. Our fan sits on top of the CPU, jutting out over the memory. The memory stick I ordered has a cover that forces the fan to move up 5mm. It is not a big deal, but you can order some other memory and save the hassle.

CPU cooler

As I mentioned earlier, the usual non-overclocked version of CPU comes with a fan. I went for the overclocked version only because there was a deal. I ended up having to order a special fan.

What is the deal with a fan? I did not find many differences. For the work we do, for the climate I am in, for the case I have, heat is not a big problem. The case itself has lot of fans too. The higher end ones with liquid cooling are not worth the trouble for me.


The cooler master Hyper 212 Evo has a large heat sink. It comes with the thermal compound to place the heatsink on the CPU. It has all the attachments required. If you find something good enough on the sale, it should not make much difference.

Hard disk

For my need, the choice is between m2 SSD or SATA SSD. The SATA SSD is like a hard disk. The M2 SSD looks like a memory card and sits in one of the slots. It is really fast, but the price is too much for the budget. So, I went with a 500 GB Samsung SSD.  This SSD may require some additional hardware to place in the case. But, really speaking it is not needed as long as it is plugged in and placed without the danger of falling down in the case. You can you use a rubber band, for all you care, to fix it in the case. Or, spend $5 and get the attachment to place it in 3.5 bay.


How to buy

Pcpartpicker does the price comparison and tells you where each component is cheaper. Be aware that you have to add taxes to your price on Amazon or your local retailer. Places like newegg or B&H do not collect taxes, though technically you should pay the taxes by yourself end of the year. Surprisingly, I found www.microcenter.com to be the cheapest. It has a retail store in my neighborhood. I managed to pick up the case in open case price, making it really cheap. The CPU+Mobo was on a combo price deal, saving me another $40. Overall, it was cheaper by $120 — a good 6% than the online companies.

Order all the parts at once so that you can start building immediately. In case there is a problem, it becomes clear immediately. You want to take advantage of one month return policy most merchants have — so, start building immediately.


I never built a computer. Thanks to Youtube, I found it easy. Here are the videos I used:
https://www.youtube.com/watch?v=T9pShHlCmLc&list=PLc3lzolCKqOOnVDfa6OpbuYX2f28R-31x — this is a three part video series. It makes it really easy to build the PC. If you want a quick no-nonsense video: https://www.youtube.com/watch?v=0bUghCx9iso&t=1428s

In any case, here are the steps:

  1. Install the CPU
  2. Install the CPU fan
  3. Install the memory
  4. Install the motherboard into the case, taking care the back plate is placed snugly.
  5. Plug the case fans into the motherboard
  6. Install the graphics card
  7. Install the hard disk
  8. Plug the front panel cables into the motherboard
  9. Put the PSU in.
  10. Plug the power into the motherboard
  11. Plug the power into the CPU
  12. Plug the power into GPU
  13. Plug the power into the SSD and the front panel (and DVD drive, if you have one).
  14. Arrange the cables neatly.
  15. Now, test the case and close the case.

I am not going to describe these steps in detail, as the videos are very descriptive.

After building the computer, check the BIOS. These are the steps you want to do:

  1. Check if the BIOS recognized all the parts.
  2. Update the bios, if there is a newer bios.
  3. Install the OS.
  4. Update the clock speeds.

I am going to describe installing OS only in a little bit of detail, as it may be instructive:

Installing OS

As you are plugging in the peripherals, you will notice that your monitor can be plugged into the motherboard or the graphics card. By default, if the card is there, you should plug into the graphics card.
If you are installing using the flash drive, you would need to change the booting order. There are some issues installing Linux desktop with XWindows. Instead, my suggestion is to install Linux server — which skips the graphical installation entirely. You can the graphical installation after installing the drivers. You can use rufus (https://rufus.akeo.ie/) on windows to copy the ubuntu ISO to a bootable flash drive.

Once you install it, my suggestion is to follow https://blog.nelsonliu.me/2017/04/29/installing-and-updating-gtx-1080-ti-cuda-drivers-on-ubuntu/ — While I am not able to independently test installation using other methods, this methods worked on my machine.

How does it fare?

I will tell you the full details later. But, for 2K, I have a GPU server for which I need to pay 70c per hr. Assuming that I run it for 8hrs, it is roughly $150 per month. At that rate, it pays for itself in an year and half. Of course, I can run bitcoin mining, and make some real money!

Next part

In the next part, I will tell you how to start with your GPU machine, basic setup, basic tools, and basic steps in GPU computing.

Feb 092017

Let me ask a question: name one south Korean computer science professor that is big all over the world? Or, even manager? Quick. I could not. On the other hand, name some Indian scientists or leaders. You can name several. But, name on Indian company that is world class in innovation? None come to mind as readily as Samsung, a South Korean company.

The point is this: in India, we optimized the systems for producing small number of excellent winners. In South Korea, (and most Scandinavian countries, for that matter) they optimized to improve the average people. In the book “No full stop in India”, Mark Tully writes about this phenomenon: even the successful Indian industries are geared towards single heroes, as opposed to large teams. India is the land of artisan that work by themselves. It is common to see a blacksmith in the village working all by himself.

The software industry in India may have started out by celebrating the individual heroes, but over the years, come to value teamwork. Most of the people who joined these companies are college grads that had limited exposure to programming. The success of these companies to bring them together and create a team and make them specialize in some computer tool, or programming paradigm of the times.

2476For instance, when SAP was vogue, they trained people on SAP. When Java was popular, suddenly you have lot of J2EE programmers. Now you see lot of places training people on DevOps, or even machine learning. It is an adaptive market.

In the end, what these companies aspired to do is to industrialize the development. They take the moniker “assembly line development” seriously.  These developers know their tools well; they understand the process of getting the requirements and translating into code; they understand the abstractions they operate on — that is, somebody already decided on the entire stack, designed the framework, and handed out the module names for these people to code.

As enterprise developers, they are cocooned from some aspects of development. They do not think about where the programs are run. That is somebody else’s worry. They do not think about operations. Again, somebody else should take care of that. They do not do integration testing, leaving it to the testing group. They don’t do performance testing or DR testing either.

They do not have freedom to use what they want to either. They cannot use a labor saving library, because of the complexity of managing a new piece. They cannot choose the database for their solution — they are provisioned on the ubiquitous relational database with JDBC or ODBC connectivity. In general, enterprises are not a friendly place for open source, often with a good reason.

In the end, they became like a glorified assembly line worker, in a non-fully automated assembly line; and that too in a work where creativity is often required.

Modern application developers are different. They operate on full stack. They experiment and come up with creative designs.


On the first blush, it seems problematic. Isn’t it inefficient? Didn’t economics teach us specialization improves productivity? Isn’t the whole innovation of industrial age is towards specialization?

There are reasons why it is not. Here are some:

  1. Firstly, the current state of modern development is not suitable industrialization. The tools are changing constantly. The platforms are changing too. Any attempts at industrialization leads to suboptimal solution. In fact, that is where the Indian industry is stuck — a suboptimal solution from ages ago.

  2. Secondly, the tools that the modern developers use automate large number of tasks. The modern tools make the modern developers short circuit on lot of the tasks that only exist in the enterprise world. For example, the overhead associated with release cycles? Gone. Or, even the requirement delays? Gone.

  3. Thirdly, the modern development offers a way to optimize the whole process, as opposed to individual parts of the process. For example, consider deployment engineering. Typically, it is done by different team that has no input into development. Without cross-team collaboration, there is no way we can optimize the processes. With full stack development, the small teams understand each other well. Better yet, the same team may be involved in rolling it out on the platforms designed by other teams.

Of course, there are others too, but in my opinion, the modern stack development works despite the broad based knowledge it requires of its practitioners.

What should companies do?

Companies see this phenomenon and interpret it differently.

They see that startups, silicon valley developers create applications differently. They see those people be more productive and more agile. They end up believing in the myth of super programmer. They think that they can get these super programmers to impress the customers. They think they can get these programmers to change the whole organization.


Lot of companies — I do not mean merely Indian companies, but the non-valley, non-hi-tech companies — are turning to these kind of developers. They get these people on the payroll hoping to transform the organization. But, these imported super stars or full stack developers do not understand how to change the organizations. They worked in an environment that made them productive. They cannot create such an environment in the new organization. They do not understand how to bring such a change to the new organization.

To solve this problem, companies came up with different strategies: they try bi-model IT (one part staffed with modern technologists and the other with traditional enterprise developers). They talk about pace layered architecture, where some systems are always built with modern technologies.

Yet, none of them work without creating a change beyond a few hires. They need to create a scalable model to create such developers.

My advice to the Indian companies is not to fall for such traps as hiring a few super programmers. Instead, they should increase the averages, like I said in the beginning.

What does it mean? They should focus on creating a new kind of environment, starting with the way they organize the teams. We cannot have multilayered system, where managers are there only to direct others to do their work. Like the HBR article said, the manager should be able to do the job of his reportee. That should cut down on the layers a bit, I would think.

They should focus on creating a collaborative structure, that is beyond command and control. It should be unstructured; it should be context aware; it should be ad hoc; it should promote creativity and teamwork.

They should create a new training regime. Do not teach what is there on the internet. Let them learn it by themselves. I find the following ways work well, based my experience with one of my startups:

  1. Set expectations: The old saying is practice makes perfect. No, it does not. Practice makes permanent. Learn to do it wrong, and you will do it wrong forever (like my typing). Instead, establish the best practices upfront.
  2. Make them learn on the internet by practicing on the open source: Have them contribute to the open source. It can be documentation; it can be testing, porting or, a minor feature. That teaches them the basics are distributed programming.
  3. Make them practice with the real problems they face in the world: Instead of starting a problem from the beginning, have them take a large problem and solve a bit of it. It makes them read and understand the code. If we choose the right problem and right solution, they will learn from the masters.
  4. Let them focus on basic tools: We are enamored with large tools. We see if people know how to use a large system, say, Hybris commerce server, but not how DNS works. Lack of knowledge of the basics will hinder them where they are asked to fill the gaps in the state of the art platform.
  5. Create a checklist before they can join a program:  This list can validate if they know the best practices. Let them be opinionated. Let them learn to defend their positions. As long as they understand all the basics, they can contribute to the project.

In the end, a company that can transform its workforce to improve the average will do wonders. Superstars are for startups. Enterprises still need their large workforce until AI can write the programs based on what we talk to it. That will happen someday, thanks to the super programmers. Until then, heed my words!

 Posted by at 9:50 pm
Aug 232016

Recently, I was asked a question: how do you create a sustainable knowledge management? The answers that they previously got apparently were on these lines:  of different knowledge management systems, the processes, the governance, the artifacts they would produce and the structure of the dedicated team that works on this knowledge management. Mind you, this was not a project on knowledge management, but a program that the client was getting into. They thought that they would knowledge management for that.

There is nothing to blame the answer. Years of management learning taught us that if we want to solve a problem, hire people to do it, allocate resources to it, and show commitment to it. In fact, we talk about how do “organizational change management” (OCM).

Sure enough that there are examples of such cases. Famously, IBM showed how it can create new systems and support it. Microsoft reinvented itself in the internet era – by doing exactly the same, in creating a new focused division.

Yet, in this case, this answer is entirely wrong. Unless the company is in the business of knowledge selling, knowledge management systems will fail. This kind of approach fails when the focus in not meant to be knowledge management. This problem is particular in lot of knowledge sharing applications, collaboration systems, training systems, and so on.

The general explanation that people give is that the designers failed in creating the right programs. That they focused only on computer systems and neglected OCM. Or, that they did not allocate enough money for support.

(© Scott Adams)

All of these reasons are correct, yet not really completely correct. There is a reason why lot more money is not allocated – the management does not see the ROI. Or, there is a reason that they neglected OCM – this was not the focus.

So, do we say that these systems are not important? Can we make them important? Or, should we even make them important?

Let us look at this issue in depth. As organizations are trying to become digital, they recognize the importance of these kind of supporting systems and yet do not know how to make them succeed. They work about bringing lot more consultants to change the organizational culture and continue to fail. Therefore, it deserves more in-depth understanding.

Theory of sustainability

In the beginning, in the question I was asked, there was a key word:  “sustainable”. We normally see that word in ecology, architecture, and social studies. There, people talk about enduring systems, systems that replenish themselves, systems that do not require external stimulus to support themselves.  Normally, in ecology, people talk about the footprint of the systems: the effort it takes to create a system, maintain a system and so on. To make a system sustainable, we want to make sure that it has low footprint, and it does not depend on somebody’s explicit effort to keep it going.

For example, let us look at sustainable architecture: it means the footprint of the architecture is low. It can sustain itself – perhaps through low energy usage and perhaps with solar power. Also, it may recycle water. It may reduce wastage. All of these metrics mean that it does not take much to maintain the architecture. For example, in my village, these thatched houses are built with crossflow ventilation and open spaces. These days, the modern houses are built with bricks with window A/C units. Come summer power cuts, the modern houses are unbearable, whereas the old houses are merely uncomfortable. It is same with urban design also, when the it is harmonious with nature, it is easy to sustain. Most cities cannot sustain themselves without lot of effort these days into the roads, parking lots, public transportation, sanitation, and water services. Imagine that we build cities for people! We may be able to make them sustainable.


[See the contrast between two designs: the first one relies on parking spaces – it needs more and more parking space as it needs more and more space and only cars can traverse the distance. The second one is built for people, where walking is the way to get around. It works because it needs less space. Pictures courtesy: http://www.andrewalexanderprice.com/]

Surely, we can take lessons from these systems in designing our knowledge management system? Designing for people with low maintenance or no maintenance? Or, working with the way people do, instead of trying to change them? Working with the existing culture instead of imposing a new culture? Making “nudge” like social experiments instead of, say great leap forward?

These are hard questions, with no definite answers. Yet, we can make an attempt at understanding the role of sustainable systems in IT and key elements to creating such systems.

Three kinds of IT systems

Let us say that a bank is getting into new business of serving mass affluence customers. Historically, they served only the customers with 10 million dollars or more. Now, they want to extend the same services to people with 250K. Let us say that the organization is building a new division to support this initiative.

Even if it sounds easy, this is a difficult task. Firstly, the employees are used to handhold the millionaires. They are used to bypass the processes to cater to the super rich. Now, that they are working with regular rich folks, their old ways cannot scale to the new numbers. And, their compensation earlier may depend of different metrics: handling a few HNI. Now, they have to use different metrics – may be total money under management. The systems they use are going to designed for most common used.

Let us confine ourselves to the computer systems (and supporting processes) to run this division:


Let us see what these systems do and where sustainability may make a big difference.

Core systems

Without these core systems, the business doesn’t even run. The management makes sure that the focus is on these systems. These systems run with stringent SLAs, with adequate funding, and often with milestones, bonuses, and penalties. These metrics are very closely aligned to business metrics. These systems do not need to be sustainable – there is enough support to sustain them by providing external stimulus.

However, there can be a case to make the architecture of these systems sustainable. Often these systems are rewritten to support changes in business. These rewritings can be expensive and may cause delays in the ability to respond to the markets. If we make the architecture sustainable, then we can make changes to the system without causing major changes to the core part of the architecture. For instance, if we have good modularization, good interfaces, and coherent philosophy of data, we can extend the existing architecture without any changes. For instance, see how Unix endured for the last 50 years, without fundamental changes.

This kind of “built-for-change” architecture can make core systems resilient to change – and can create sustainable architecture.

Required support systems

The required support systems are not part of the core systems. But, to run the core systems we need these systems. The metrics on these systems, the SLAs and how well they are run, do not directly translate to the main core metrics. Yet, they are important, like insurance. For instance, if security is a problem, you would hear about it. If operations is a problem, you would again hear about it. If there are no problems, the chances are you won’t hear about these systems at all.

One interesting aspect is that these systems are very similar across different businesses. For example, would the HR be that different for a bank, from say, a retail company? The chances are that they can get away with using the computer systems that are generic – say software as a service in the cloud. Or, even better yet, they can even outsource most of these functions to outside. Cloud fits in here very well, either as a full end-to-end service offering or a partial service offering.

(from: http://founderdating.com/marketing-changing-saas/).

Since these systems are mandatory, even if there is no direct business value, IT is often measured on these systems. Because of this reason, these systems do not need to be self-sustaining. Still, since these are not core to the business, very rarely do companies want to innovate in these systems.

Optional support systems

These systems are what management talks about in the vision. For example, when a company says that employees are the best assets, these are the systems supposed to increase the value of that asset. Every visionary program talks about how these systems are created and maintained.

And, yet, these are the systems that end up failing often.

Since these are not in the way of core delivery, even if these systems fail, nobody raises alarms. They are like the climate change harbingers. A few polar bears dying? Not a mainstream concern! Yet, we all know that without these systems, modern digital enterprise cannot survive.

Why did I make such a bold statement? Take a look at the models of the modern digital enterprise: Google, Facebook, Netflix and such. Almost all of these place a lot of emphasis on hiring, creating, and nurturing talent. They take these optional systems and turn them to competitive assets so much so that most companies that want to be digital, want to emulate them on these aspects. The whole idea of innovation labs or devops, or 2-speed IT, or data driven organizations take these practices as a way of creating such dynamic organizations.

For an organization that wants to see these optional support systems as a core strength need not look at digital enterprises alone. They can look at open source ecosystem, where these optional systems are well developed and sustained. For instance, in open source world, if a volunteer joins a new software development effort, how does he go about it?

  1. He can read the code and documentation on github.
  2. He can read the old notes or discussions on mailing list archives or google groups.
  3. He can participate on the slack channels.
  4. He can fork the code
  5. He can fix some simple issues and contribute back
  6. He can enhance the documentation.


In short, without taxing any resources, he can become a productive member of the team. In fact, these are the salient features of this development process that makes it sustainable:

  • All the optional systems are baked into the required systems: the knowledge management is a part of version control system.
  • The optional systems are simple. For instance, most of the documentation is written in markdown. It is a part of the source code. It is not elaborate, but quick to the point of boot strapping.
  • It welcomes contributors. The process of forking and contributing code is simple. In fact, the first think people handle is this simplification, before they set the project up in github.
  • If the leadership changes, the entire project can be cloned. There is no hidden information.

In contrast, in an enterprise, this is how it might be:

  • A local, special toolset that forces people to learn before they contribute
  • A special team who is supposed to maintain onboarding documentation, which often is out-of-date,
  • If one has to contribute to the project, it will take handholding of an existing (often busy) team member
  • No quick gratification for the new comer – which means there is no incentive for new comers to join the program.

In short, the open source way is sustainable because it is simple, it is welcoming to new comers so that they can sustain the system.

Sustainable organizations

The same principles work in creating organizations as well. Most organizations focus on customers and deliverables, which contribute to the direct metrics. The indirect metrics that enable these metrics in the long term may be focus for a while, but the moment they become less important, they fall by the way side. Because they are not sustainable, there is every danger of them becoming a wasteland of systems with lip service.

Most organizations do realize this problem. Their solution is to increase the focus on this problem. But, that does not solve the problem forever. Instead, it becomes like US medical healthcare. The joke is that people with rich plans end up getting bad treatment, as everybody wants to treat them for something. And, people with no insurance also end up getting bad treatment. Only with just the right kind of insurance, will we get the right treatment.


By Arshiya Urveeja Bose – Flickr, CC BY 2.0

The idea is the focus should be creating such sustainable systems – the systems that can run without continued focus. These systems should run, as a byproduct of the main metrics. They should run because the traditions established. They should run because they serve selfish interests of people. They should run because the collective interests of the people to force others to confirm.

Checklist for sustainable systems

We started with the knowledge management system and how to make it sustainable. I never gave any answer to that question. Instead, I went on to explaining sustainability. Now, let us get back to sustainable knowledge management system. While we cannot define it, we may be able to help in creating a checklist to see if a system is sustainable. Obviously, we want it to be of low footprint, but that is not enough.  The following check list may be a good start.

  1. In the steady state the system should not require any additional resources to support it.
  2. Until it gets to the steady state, the stakeholders should commit to it.
  3. The system should automatically keep itself relevant.
  4. It should be in the interests of an individual to use it.
  5. It should be in the interests of an individual to contribute to it.
  6. It should be in the interests of the team to use and maintain it.
  7. It should be easy to get started with the system.
  8. It should not contradict existing modes of working.

(© Scott Adams)

These kind of systems can be sustained in the long run. If we are building a sustainable knowledge management, this is the list we should check against.

Or, go with knowledge optimization initiatives to leverage our key learnings, as Dlibert says Smile.

 Posted by at 10:03 pm
Apr 072015

Every company claims to be customer centric these days. But, then, if you look back, companies always claimed to be customer centric. “We are focused on what the customer wants, right?” They say. In fact, it was and is the bed rock of American industry, we are told.

All true. In deed, organizations have been focused on the customers always. However, when it comes to IT, the story is not so straight forward. Consider the following evolution of IT.


Let us look into a brief background of this evolution, with some examples.

System Centric IT

As organizations started going beyond payroll systems to using computers in running the business, the primary concern is about the functionality of the applications. To large extend, the PCs of that era also influenced the purchase decisions: We want as many tick marks as possible in features.

Consider the IT manager trying to automate procurement. Any procurement application requires partner qualifications, tender management, contract management, PO generation, quote generation, etc. (I am not an expert in this area, but you get the gist). The IT manager looks at various products and makes a judgment based on several factors, including functional fitment. That is the how IT is built of several systems, selected for different functional capabilities.

What is the flipside of this kind of selection? Let us look at the problems and the ad-hoc solutions that IT had to deal with because of the fragmented systems in IT.

Overlapping functionality

The overlapping functionality of the systems in the enterprise leads to lot silly problems that IT vendors have been making money off for several years:


Of course, there is a simple solution: Buy one system that solves every problem of running a company. The big ERP systems (an excellent example would be SAP) promises exactly that. You get a unified view of the world, with one application (even if it is broken as modules) dealing with standardized data, clear delineation of responsibilities, and best practices about what functionality is to be implemented in what module.

Despite all these promises, enterprises still will have to build applications, buy different applications for different functionalities, and live with the consequences.

Process Centric IT

System perspective of IT enables an IT to automate all the core functions. Yet, when we look at an enterprise, more and more, the business sees itself as carrying out some processes. For example, procurement is a process, managed by a business unit.

This process view in a company has several benefits: firstly, companies can measure themselves by process. For instance, how long does it take to procure a new computer for a company? If that is the metric that needs to be lowered, then, people can modify the process, optimize the process, or simply train the people to process requests faster. Secondly, process is natural to companies, as they are optimizing for operations. Since operational responsibilities fall along the lines of process, it is convenient to create KPI’s based on the process.

In the world of IT that is built on systems, this process centricity poses a problem. Let us look at the ways:

  1. If a process is entirely contained in a single application, it looks almost like a functionality, right? Then, what is the problem? The problem is process view demands certain uniform approach to process management. For example:
    1. Process visibility: Where exactly is that request for new onboarding of a supplier stuck?
    2. Process metrics: How long does it take to process this new supplier on-boarding?
    3. Process delegation: Jane went on vacation. Can John who reports to her approve the request?
    4. Process allocation: We have three people verifying the supplier bank information. That is the where the lag is. Can we increase the number of people handling that task?
    5. Exceptional processing: We have an important project for which we need this new supplier. Can we hand-hold the process to move it past the gate?
  2. If a process spans multiple applications, that creates additional problems. For example:
    1. All the process management requirements become that much harder to implement. For example, consider delegation: You may have to repeat the same delegation in two different systems.
    2. Integrating the processes is complex. We need to bring the data in sync between different applications.
    3. Process optimization is complex. It is difficult to change two systems and keep them in-sync when the overall process is changing.
  3. If a process is not fully automated, it creates even more problems:
    1. Process management may be impossible, with manual steps. In particular, visibility, metrics, and allocation – all have to be done manually, leading to inconsistent results.
    2. Process operations may be costly, because of the manual steps. In addition, it may be error prone and inconsistent.

With all these problems with systems view of IT in a process world, what is the solution?

BPM for process management in a company

BPM as a emerged as a general mechanism for business process management in an enterprise.


Consider the implications of the picture presented:

  1. It is preferable to implement a process outside of the systems, in the BPM, if we are looking to manage effectively.
  2. Even if the process is contained one application, it still may be preferable to implement in BPM.
  3. BPM can provide additional benefits, in addition to process management:
    1. customization of process, which may be simpler do in BPM than the application
    2. Reconciling different data models – customers, partners, orders etc.
    3. Automating simple tasks, without being custom systems – to full automation of the process.

The enabling technology is SOA, which lets applications work with a BPM system.

People Centric IT

Process centric view is fine for lot of purposes. It lets IT support efficiency drive in enterprises. It lets the companies focus on the processes that matter most. In fact, it gives the organizations a blue print on understanding which processes to customize and which processes to leave alone.

But, one important aspect that gets hidden is this: How do people use the systems? Are they happy to use the systems? Do these systems work in the context of the usage (like, say on the go, or in the office)?

As an analogy, consider a government office, say DMV. If you are there for license renewals, you stand in one line. If you are there for registrations, another one. Auto titles, yet another one. That is, users are tasked with figuring out where they should stand.

Much like that, most of processes typically have their own way of serving users. Even if we were to integrate them in one portal, the experience is different. In fact, users may be forced to learn lot of different interfaces depending on the context.

Or, consider a corporate website. Typically they are designed describing the way the company is organized.  But, a visitor does not care about how the company is organized – they only care about how their interactions are organized.

When we start taking a user centric approach to IT, we should not expose the systems or processes to users. After all, systems are the way vendors developed the applications, and processes are the way organizations carry out their work. We should design the interactions of users with the internal systems and processes and focus on providing the user interface supporting the context of the usage.

What customer centricity means to IT

If a company decides to reshape their IT to incorporate customer centricity, what should it do?

First and foremost, it should realize that the most of the systems that it can acquire still will be operating in the old-fashioned mode. And, internal processes will remain the same. Until products and internal applications are retooled to customer centricity, we will have to implement customer centricity differently.

Designing for one

Ideally speaking, we should design an application for every user. After all, every user is different in the way they want to use the systems. But then, it is too costly for that level of customization even with personalized applications.

The right approach to that is through Extreme Segmentation. That is, we do not divide people into segments based on static attributes. Gone are the days when people are classified into established buckets such as “college educated over 50’s white male”. Instead, we provide several attributes of users that lets us group users into very many overlapping sets.

To support that many segments, we may want to retool the IT in the following way:

  1. Self-provisioning, community-supported, discoverable applications
  2. Easy to develop, deploy, and operate applications
  3. Standardized applications catering to segment specific interactions.

I will be writing more about extreme segmentation in digital marketing soon.

Three main groups of users

While it is nice to talk about extreme segmentation, I find the following three groups of the users very important:


Each group of users use the IT systems differently. And, as such customer centricity would mean different for each group. We can consider other groups such as partners, operations people etc – their needs tend to be a combination of these three groups.

Standard approach to customer centric applications

Finally, let me summarize by describing a standard reference architecture that can be used for developing customer centric applications.


This is somewhat simplistic, but the essential principles are going to be same even in a full blown picture. In summary, the architecture uses:

  1. API’s to build the experience on top of existing applications
  2. Use modern UI technologies to design applications
  3. Applications can replicate functionality, in catering to different sets of users.

Concluding remarks

In this note, we discussed the emergence of customer centricity in IT. We described what it means to IT and architecture. In subsequent notes, I will describe the following:

  1. What extreme segmentation means and how it works in digital marketing
  2. A prescriptive approach to creating a customer centric IT – an enterprise architecture approach
  3. An in-depth look at architecting a customer centric application: concept, design, and implementation

Till then, happy reading!

Mar 172015

What if I were to tell you that the e-commerce site you’ve been developing could be hosted on a static website that would offer it premier performance? Or that the document management system could be run on any plain, old static server like Apache, nginx? Stick with me and we’ll explore this topic. This information may be too basic for advanced developers, but for most managers, this introduction may be long overdue.

A bit of history

Long ago, most websites were static. All they served were pages – html, text, and images. Hyperlinks allowed us to navigate from one page to another. Most were content-rich websites containing information, entertainment, and photos. Now it seems unthinkable that clicking on the hyperlinks was the only way to interact with the sites.


As shown in the picture, the protocol between the web server and the user is fixed: http protocol. The way the webserver interacts with backend information (in the case of static web, it is html, css, js, images and such) is up to us. A static website serves this information straight out of the file system.

As the readers began wanting to navigate sites in interesting ways or interact by adding content with the websites (like comments), we began to need server side programming. Serving out of a file system won’t do. The application needs to understand complex requests and somehow use the information in the files (or databases) to serve the information. Since database is a popular choice to keep structured information, web started serving as interfaces to the database – types of simple crud operations. In fact, applications like phpMyAdmin even let people manage the databases on the web.

Coming back to the web applications I mentioned, — e-commerce, digital marketing, document management systems — all these websites are considered interactive, dynamic, and programmable, and therefore need a web application server. The moment you choose one platform (typically it means an app server, a language, a framework, a set of libraries), you’re stuck with it forever.

Let’s see how we can design our way out.

Static web with automated look and feel support

Let us consider a simple case of information presentation. In this setup, all the consumers want to do is read the information. In that case, why not stick to a static website?

We can. The advantages are obvious. First and foremost, a static website is simple. It does not require any moving parts. Any old webserver can serve static content. You can even run a static website, with a simple one-liner in Python (python –mSimpleHTTPServer in Python 2.7 or python3 –m http.server).

It is also incredibly fast. The server has to simply send pages – it does not have to do any other computation. As such, it can be sped up even more by caching the content locally, or even using CDN (content delivery networks). You can use special purpose webservers such as nginx, that are optimized for serving static content too.

Furthermore, it is simple to maintain. You can back it up in the cloud, move to a different provider, and even serve it from your own desktop. You do not need to back up a database or worry about software upgrades, portability issues, or even support.


But, what are the downsides? Let us consider the problems:

  1. How do you create nice looking pages? Let us say that you want to create a standard look and feel. If you are the kind that use hand-coded HTML, you will cut and paste the HTML template and then edit it. Pretty soon, it becomes complex. If you want to change the look and feel, you will have to go through all the pages and edit the few lines.
  2. How do you create standard elements of a website? For example, you want a menu scheme. If you add a new area, then you have to go to each page and add a link in the menu to this new area of website. Or, if you want to create a sitemap, you will find yourself constantly updating the sitemap.

Considering that a website has repetitive information depending on the context, static way of maintaining a website is cumbersome. That is why, even for the static websites, people prefer WordPress, Joomla, or Drupal – all of which are meant for dynamically creating websites.


In these systems of dynamic content generation, the content is generated based on the request. No matter what application you use, it will have following issues:

  1. The application becomes slow: The server needs to execute the program to generate a page and serve the page. Compare it to the earlier scenario where the server merely had to read the file from the file system and serve! In fact, the server can even cache it, if the file doesn’t change often.
  2. The application becomes complex: All the flexibility of combining content with the themes based on configuration comes at a price. The application server has to do many things.

Consider the following full description of the stack. imageAny web server that is generating pages dynamically, needs to depend on an execution engine. It can be an engine that interprets a language, or executes a binary. For any language, we need a framework. For instance, if we are using Javascript, we can use Express as the framework. This framework can do many things. At a bare minimum, it can route the calls – that is, it interprets any request and maps the request to the right response. Next, it needs to have libraries that deal with writing the logic to actually manipulate the data. To present the data, we need a templating engine.

Of course, you have fancy names like Model (the data), view (the templates etc), and the controller (the routing) for this kind of framework (MVC).

The problem with any stack is that once you are committed to it, it is incredibly difficult to change it. To make matters worse, the frameworks that let you start easily (for example, lot of php based frameworks) are not the best frameworks for enterprise strength requirements (like say Java, which is excellent in libraries that deal with lot of integration). Ask Facebook! They started with Php and had to lot of optimizations to get the performance they want.

How can we do better? Can we still use the static website, and support better usability of the website? Reduce the costs of maintenance? The answer is yes. If we were to separate the content generation and content delivery, we can get best of the both worlds.


From this activity, what did we gain?

  1. We gained performance: The runtime has to serve static files, giving us several different options to improve the performance.
  2. We gained simplicity, somewhat: Technically, it is possible to mix and match different systems of generation (which is somewhat like using different languages and compiling down to machine code).

Readers can observe that content delivery networks can do this job of caching generated content on-the-fly, giving us the illusion of static generation of content.

Static website generators

If we are going to generate static website, what options do we have? Let us see the alternatives.

  1. Firstly, we can use the same dynamic content generators and dump out the entire site in static pages.  That is, if we are using Drupal, we can dump all the nodes. Or, with WordPress, or any other content management site.

    The challenge is, what if there is no unique URL for a content? Lot of content management sites offer multiple ways of accessing the static content. For example, Lotus notes was notorious for such URL generation. Then, the static content dumping can be difficult. More over, these systems are not meant for static website generation – the limitations keep showing up as you start relying on them.

  2. Secondly, we can use WYSIWYG editors such as Dreamweaver. They can create static content, apply a theme, and generate the website. They come with several beautiful themes, icons, and support scripts as well.

    The challenge is that these systems are not programmable. Suppose you are generating content from external system. These systems do not provide a way to automate the content ingestion and upgrading of the website.

  3. Thirdly, we can use a programmable system that generates the content. These days, this is the favored approach. These systems generate or update the complete website just from text based files. You can programmatically import content, update the website and publish it to production – all through scripting. Furthermore, they offer standard templating, support for CSS and Javascript libraries.

    The downside, of course, is that these systems are not mature. They are not general purpose either. Still, for an experienced programmer, this is a wonderful option.

There are several examples of  the third type of generation systems. The most popular ones are the ones that support blogging. For instance, Jekyll is a popular static website generator, written in Ruby, with a blog flavor. The content is authored in markdown format. Octopress is built on Jekyll supporting blogs.  In Javascript world, there are blacksmith, docpad, and a few more.

Out of all the contenders, for my use, I like hugo and docpad. Hugo is the simplest of the lot and extremely fast. Docpad is more versatile in supporting lot of different kind of formats, templates, and plugins.  In hugo, all that I had to do was to create a hierarchy and drop in .md files as description. Based on the folder structure, it creates the menus, content layout, and the URLs. Docpad is a bit more complex, but it is also essentially the same.

Static web with high interactivity

There is a big problem with the earlier approach. Consider the example we were giving about a document management system: what if we want to search for the right document? Or, sort the documents by date? Or, slice and dice the material? Or, draw graphs, based on the keywords?

For all these tasks, historically, we depended on web server doing all the heavy lifting. Do you want to sort the documents? Send a request to the server. Do you want to filter the documents? Send another request.


While this kind of interaction between the user and the web server gets the job done, it is not terribly efficient. It is a burden on the server; it increases bandwidth requirement; it feels sluggish to the user.

Thankfully, the situation changed with advancement of Javascript. Turns out when any HTML page is digested by the the browser, it creates a data structure called DOM. With Javascript, you can query and change the DOM. That means, at run time, you can do sort, filter, display, hide, animate, draw graphs – all that with the information available to the browser.

With this kind of power, we can design interactive websites without going back to the server. With this additional requirement, how we develop web pages and what technologies we use will be different.


See the sequence diagram titled “JS based interactive sites”. The web server sends the data, html, and javascript the first time. After that, any interaction is handled by the script itself. It can compute the new content based on the user interaction, from the existing data and make the necessary modification to the page by changing the elements of DOM.

The implications of this design option are enormous. We do not need to burden the server; we do not need to overload the network; we provide quick interaction to the customer.

The range of interactions is limitless. For instance, if we are offering an electronic catalogue of books, we can search the titles, sort by authors, filter by publishing date, and so on.

In fact, these kinds of interactions are common enough to have several mature libraries supporting them. For example, for the tasks I mentioned in the previous paragraph, I would use dataTables. If I am doing the same with millions of entries, I would use Pourover by NYTimes (which used this library for their oscar award fashion slicing and dicing web page).

Static web for collaboration

For the kind of interactivity we discussed earlier, Javascript libraries work well. If you think about it, all those web pages are providing is read-only interactivity. What if you want read/write interactivity?

For example, if you have a website with lots of content. You want to provide people a way of adding comments. Take the case of earlier Octopress itself – we may want to add commenting capability to those blog posts. How would we do that? We certainly need server side capability to do that.

Fortunately, the server-side capability need not come from us. We can easily use 3rd party server for these capabilities. For instance, we can use disqus or Facebook for comments. We can use Google analytics to track the users.


In fact, the 3rd party server ecosystem is so strong these days, we can develop highly interactive websites, with just static content served out of our server. You can learn what other leading web companies are using on their web pages and what services they are using from http://stackshare.io/.

For example, if you want to use payment services on your website, what kind of 3rd party service can you use? Look at the choices: http://stackshare.io/payment-services: Stripe, paypal, Braintree, Recurly, Zuora, Chargify, killbill and so many others.

How do you integrate into your website? Here is an example:

<form action="/charge" method="POST">
    src="https://checkout.stripe.com/checkout.js" class="stripe-button"
    data-description="2 widgets"

This action creates a checkout button. When the user checkouts, it creates a token and provides the browser. You do need to have some server side component that takes this token and charges the user – it goes beyond the usual static website I was describing. But, for most other less critically secure website, you can conduct the updates from the browser itself.

For example, take a look at how you would integrate the analytics from Google on your website:

<!-- Google Analytics -->
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),

ga('create', 'UA-XXXX-Y', 'auto');
ga('send', 'pageview');

<!-- End Google Analytics -->

You find this code in several websites, which posts a click data to the Google server.

Admittedly, most services need server to interact with them, for security purposes. Nevertheless, the heavy lifting is done by some 3rd party web server.

Static website special case: Server less setup

A special case of static website is desktop website. Suppose you want to share a folder with your friends. Easiest way to put the folder in dropbox and share it with them. Now, suppose, you want to provide some interactivity, say, searching the files etc. What would you do? You could host the files in a website. Too much trouble. You could run a local webserver. But, that is too complex for most people. Why not run the site with file:// protocol, without requiring a server, directly opening a file in the browser?

This approach works surprisingly well. The workflow could be as easy as this:

  1. Let people, including you, place the files in the shared folder.
  2. Watch the folder on update (or, do it periodically) and run a script that generates the data file.
  3. The user can open the index.html folder, which uses the data file to create a web page.
  4. With suitable library (like datatables) the user can navigate, search and sort the files.

This is a simple poor man’s content management service. You can enhance the document authoring process to add all other kind of meta data to files so that you can make more effective slice and dice operations of the folder.

Static web for e-commerce: An exercise for you

Let us take it up one more notch. Can we design an entire e-commerce site as a static website (or, at least with minimal server components)? We want to be able to display the catalogue, let the users browse, discover, and even read/write the comments. In addition, we should let them add items to shopping cart and check them out. We may even want to add recommendations based on what you are looking at.

Now, how many items can we keep in the catalogue? Remember that images are in separate files. We only need the text data. Looking at general information, it is max 2K per item. While there is no limit to the amount of data browser can load, anecdotal evidence suggests that 256MB is a reasonable limit. So, 100,000 items can be displayed in catalogue, without problems. Remember that all this data and images can be served out of a CDN.

We can do one better. We do not have to load all the items at once. We can load parts of items, based on demand. Now, if the commerce site has different divisions, and the customer chose one of them, we only need to load that part.

If we can reduce the number of items to say 10,000 to start with, that makes it less than 20 MB, which is the size of a small video. So, it is entirely reasonable, for user experience perspective, to be able to load 20 MB for a highly interactive site.

What about other services? We can manage the cart in JavaScript. The actual checking out: payment, receipt, and communication to the backend need to be done an actual server. Anything less would make the system less secure. Anybody with knowledge of the JavaScript can easily spoof the browser – so, best not to make direct calls to the backend, from the browser that assumes any valid data from the browser. All you are doing is a providing the integration in the browser!

We can think of some more modifications. What if we design a mobile application? We only need to ship the deltas in the catalogue. After choosing the catalogue, the application can send a request to fulfillment with some additional credit card information.

Now, go ahead do the following tasks:

  1. Draw the technical architecture diagram
  2. Create a data model and generate sample data in JSON
  3. Create a set of javascript programs to support the following activities
    1. users browsing the catalogue
    2. Adding items to the cart
    3. Checking out cart (here, you might some server components – do it an a different server).
    4. Searching the catalogue
    5. Managing the cart
  4. For additional credit, do the following
    1. Cross-selling and upselling (people who bought what you looked for also bought the following; or, buy this additional item at a discount). Discuss the security implications.
    2. Develop a mobile application that caches the entire catalogue on the device. Figure out a way to keep the catalogue data synchronized on demand.

Have fun!

Dec 312014

Because of Facebook, I have been in constant touch with friends, acquaintances and even people that I did not meet. I am using this annual letter as a way of summarizing, introspecting, and filling in the gaps in my usual communications about technologies to friends. It is heavily slanted towards technology, not the usual intersection of business and technology.

There are three ways that I learn about technologies. One is by experimenting on my own. By actually coding, practicing, verifying hunches, validating ideas, and playing, I learn a bit. By talking to customers, sales people, engineering managers, and developers, I understand what the problems of application of technologies are. By reading books, news papers, and blogs, I try to see the interrelationships between different subjects and the technology influences in the modern world. Let me review from the annual perspective, what these three different influences taught me.

(Cloud) Container based technologies

I played quite a bit with container based technologies. Specifically, I tried various docker based tools. In fact, I setup systems on Digital Ocean that lets me create a new website, make modifications, and push to public, in less than 10 minutes. That is, using the scripts that I cobbled together (most of them are public domain, but I had to make some tweaks to suit my workflow), I can create a new docker instance, initialize with the right stack, provision reverse proxy, and push the content to that machine, install all dependencies, and start running the system.


Minimal, cloud-ready OS’s

From my experiments with docker, I see that the power of virtual machine that can run any OS is not much useful. I don’t want to think in terms of OS. What I want is a stack to program in. My OS should be cloud-aware OS. It should be minimal, should work well in a hive, should support orchestration, and should be invisible. Since I am unlikely to interactively work in it, I would place a premium on programmability, and support for REST services. Of course, it needs to secure, but since it is minimal OS, I want security at different layers.


Based on all these requirements, I see hope for CoreOS kind of OS. I don’t know if coreos is it, but something like that — a minimal, cloud ready OS is going to succeed. Companies like Google and Facebook already use such systems for their internet scale applications.

(Server side technologies) Node.js technologies

I entered this year having learnt a bit about node.js. I have a love-hate relationship (who doesn’t?) with JavaScript. On one hand, I love its ability to treat functions as first class objects, its polymorphism, its ability to extend objects etc. Its approach to type system is dynamic, flexible, and incredibly powerful.

Yet, I find lot of things to hate about JS. It scoping is infuriating. Its support for basics of large scale programming are absent. Its ability to type check are minimal. It leaves our crucial features of programming, letting users create competing idioms.

For small to medium systems that are cobbled together by REST services, node.js still is a quick way of getting things done. And, I like npm — it is even better than CPAN. I am not a fan of reinventing the wheel with all the tools like gulp, bower etc. The proliferation of these tools is confusing, putting off a casual user with their own standard tools. (Java was the original culprit in these matters.)


In Node.js, the technologies I played with are:

  • Express: Of course, duh. There may be better ones there, but I needed one standard one in my arsenal. This one will do.
  • Mongo: The power of using JavaScript across all the layers is seductive. I don’t have to translate from one format to another. And, let somebody worry about performance (well, not actually, but will do for the most part).
  • Usual set of libraries, involving, parsing, slicing and dicing, and template management.

Front end JavaScript

I have been frustrated with the front end technologies. There are too many frameworks. MVC? MVCC? And, the complex edifice makes my head swim. At the end, I am not able to make sense of it all. Thankfully, I see things changing. I am able to settle down on a few choices for myself — not because they are the best (in some cases, they are), but they are good for a generalist like me.

JQuery: I still find it the best way to manage a DOM from JS. I never bothered to learn the full DOM API, and I find JQuery convenient way of managing.

Web components: Specifically, I fell in love with Polymer. Why? Think about this. HTML has a standard set of tags. Suppose you want to introduce a new tag. You need to create JavaScript that parses the new tag and manages it for you. So, your code is strewn in a few places: the JavaScript code, CSS specifications, and the HTML. It is too messy, too unmaintainable, and more importantly, difficult to mix different styles.

Enter web components. You can create new elements as first class entries in DOM. The looks of the new element are specified via CSS in there itself. The behavior through the JavaScript also goes there. You expose specific properties and interactivity. The big thing is since it is first class element in DOM (as opposed to translated to standard elements through JS), you are able to reference it from JQuery and manage it just like you would a heading.


Since not many browsers implemented web components, we need a Polyfill, a way of mimicking the behavior. Thanks to Polymer, now we have JavaScript code that makes it appear that the browser supports this DOM behavior of web components. This polyfill intercept every call to DOM and translates appropriately.

Summary: It is slow and buggy at the moment. In time, it will take off, creating a nice 3rd party market for web components. It almost like what COM did for Microsoft.

Assortment of libs: While I did not reach my ideal setup (where the machine can speak IKWYM – “I know what you mean: language), there are several libs that help me now. Specifically, I like the templates with inheritance like nunjucks. I also use Lodash to make life simpler. And, async.js to use the async paradigm of JavaScript.

HTML looks and feel

As an early adapter of Bootstrap, I stand vindicated in pushing it for my corporate users. Now a days, almost all development we do is responsive, built on standard template driven development. Within that, I dabbled with a few design trends because I found them interesting:

  • Parallax Effect: You see pages where the background images roll slower than the text? It gives a 3D layering effect. It is particularly effective in creating narrative stories. When the images are also responsive, this 3D effect can make the web pages come alive. To take a look at some examples, see: http://www.creativebloq.com/web-design/parallax-scrolling-1131762

  • Interactive HTML pages: Imagine you are telling a story, where by changing some choices, the story changes. For examples, tabs change the content. But, imagine creating new content based on the user input, not merely showing and displaying the content. For instance, once we know the name of the reader, age and other details, it is easy to change the text to incorporate those elements. Or, if we know what they need, we can directly address in a sales dialog. While I did not carry out this idea to completion, I satisfied myself which the technology and the framework to do this interactive narrative sales tool. Naturally, this framework has a little bit of JS magic.

  • Auto generation of web pages: As I was writing text, converting the text to HTML and a suitable web page became an obsession with me. I finally settled down to using combination of md5, bootstrap, and yaml properties to generate a good looking web page to share with people.

If you are interested, please see these two blog posts, from yester years:

  1. http://www.kanneganti.com/technical/engineers-guide-to-understanding-web-design/
  2. http://www.kanneganti.com/technical/html5-design-knowledge-for-application-developers/

Static web apps

As I started seeing the advances in the web front ends, I see the possibilities of static web site. For instance, it is easy to create and run a static e-commerce application, with 10K or so SKU’s without any trouble. We can even have recommendation engine, shopping cart, various payment methods — all these thanks to web services and HTML5.


The following are the technologies that I found useful in the static websites.

  • markdown for html generation: For generic content, markdown is easy to author format. In fact, we can even author books in this format.
  • asciidoc for book authoring: For larger format HTML with more whizbangs, asciidoc is what I tried.
  • docpad for static website generation
  • pandoc for format conversions

For the in-page manipulation of large number of objects, I find the following very useful:

  • pourover: The amazing slice and dice web sites for displaying large tables is done by pourover library from NY Times. I have high hopes for it. I think there are lot of innovative uses for this library, with its performance, and ability to cache the results.

One of my favorite problems is to develop a web applications, without any db in the backend, a pure static application that acts as a library interface. We can search, slice and dice the selection using various criteria. For instance, latest document about a particular technology, written by author XXX, related to YYY document.

Mobile & Hybrid application development

I have for a long while, bet on hybrid application development, instead of native application. Now, I concede that on the higher end market, native apps have an edge that is unlikely to be equaled by hybrid applications. Still, in the hands of an average developer, hybrid platforms may be better. They are difficult to get wrong, for simple applications.


This year, I was hoping to do native application development, but never came around to it. With polymer being not yet completely ready, I dabbled very little with Angular based framework called Ionic. It was OK for the most part.

Still, for simple pattern based development, I hold lot of hope in Yeoman. For corporate contexts, one can develop various scaffoldings and tools in Yeoman generator framework. That leads to compliant applications that share the standard look and feel without expensive coding.


In my mind, right now, there are three kinds of languages: ones that run on JVM — that includes scala, Java etc. Ones that translate to Javascript: these include Typescript, Coffeescript etc. And, the rest, like Go etc. Innovation in other languages has slowed down.


Despite that, the three languages I dabbled this year are: Scala for big data applications, specifically for spark; Python, again for data applications, specifically statistical processing, and Javascript, as I mentioned earlier. I liked typescript, especially, since it has support from Visual studio. I started on R, but did not proceed much further.  Another language I played with a bit is Go, in the context of containers and deployments.

Data and databases

This year, I gave a 3 hr lecture in Singapore on bigdata, in the context of internet of things. I should have written it up in a document. The focus of that talk is what are the different areas of interest in big data are and what technologies, companies, and startups are playing in those areas.


This holidays, I experimented with Aerospike, a distributed KV database developed by my friend Srini’s company. Whatever little I tried, I loved it. It is easy to use, simple to install, and fast to boot. According to their numbers, it costs only $10 per hour to do 1 million reads per second on google compute platform. I will replicate and see how it compares against other databases like Redis and Cassandra that I am familiar with.

On the statistics front, I familiarized with basics of statistics, which is always handy. I focused on http://www.greenteapress.com/thinkbayes/thinkbayes.pdf to learn more. I followed Quora to learn about the subjects in Coursera. I wanted to get to machine learning, but that will have to wait for 2015.

On particular aspect of big data and analytics that fascinates me visualization. I played with D3 — it was of course the basis of most of the visualization advances that we see these days (http://bost.ocks.org/mike/). I am on the lookout for other toolkits Rickshaw. I will keep following it to see the new upcoming advances to make it more main stream.

Other computing tools

I find myself using the following tools:

  1. Atom for editing text files.
  2. Brackets.io for editing html files, and for live preview of the static web pages.
  3. VMWare workstation for my virtual machine needs.
  4. Markdown for note taking.
  5. Developer tools from Chrome for web development.
  6. Visual studio for web development
  7. Digital Ocean as my playground of coding activities
  8. OVH for any large size applications and servers

Customer conversations

Since most these conversations have some proprietary content, I cannot give full details here. In general, the focus in on innovation and how corporations innovate in the context of established businesses. Typically, it is a mix of technology, processes and organizational structure transformations to change the way businesses are run. I will need to talk about in byte size postings some other time.

Wish you a happy 2015! May your year be full of exciting discovery! See you in 2015!

Apr 142014

I looked at the way that the corporate training happens in technology areas and I find them wanting in several respects. This is my attempt at a bringing some best practices to corporate or institutional training.

Most corporations have training programs, especially the ones that deal with IT technologies. The goal of these trainings is to train people so that they can be useful, productive members of a project.  This is meant for training competent engineers, craftsmen who can support delivery of projects.

A typical company gets two kinds of people to train:

  1. People fresh out of college: They went through latent learning, without clear goals. They learnt different subjects in college, without clear understanding of how that knowledge is meant to be used. They tried to understand concepts without real practice.
  2. People with a little experience: They worked on few projects, churning out code. Conditions in the industry is such that they were not exposed to quality code. Most of them do not understand the value of quality not do they understand what quality is.

Current training methodology: What is wrong with it?

Typically any corporate training follows a standard process: they get nominations on who to train. They hire or get instructors, who are experts in that area. They put all of them in a room, sometimes away from all the distractions. Over the course of a day or two, the instructors will take them through (with the aid of power point), the nuances and the details of the material. For example, if the training is on Java, the students will go though the static methods, public, and annotations etc. If the training is advanced, they might even cover some patterns of usage as a way of best practices. are

Typical evaluation of students are carried out through multiple choice questions which will test the users on the intricate details of the language. These questions cover a lot of trick questions to test the understanding of the language.

What are the problems with this approach? Let me count the ways:

  1. It doesn’t train the students for what they encounter in the job. It doesn’t make them a better developer or project manager or whatever.
  2. It only tests the book knowledge, which is almost one Google query away. It is that much cheaper to invest in a good quality internet connection.
  3. After a few days of training, they forget the knowledge. Of course, they can always look up a book when they need to – but that was the situation they were in, to start with, anyways.

Even if we train them using actual small projects, these problems will continue to exist.

New way of training: What we want to achieve

The education market in the US is being continually disrupted. I am going to list a few lessons from those disruptions and later describe how to apply those lessons to our training process.


Let us see each of these lessons in turn.

Inversion of classroom and home

Known as Flip-teaching, this method of teaching became popular because of Khan Academy. The problem with class room training is that the teachers are going at lecturing at the same pace for everybody. When the students need help with homework, they are on their own.


(Image via: http://www.washington.edu/teaching/teaching-resources/flipping-the-classroom/)

In the flipped learning, the instructor doesn’t teach via lecture. There are enough number of books and videos that can teach the subject. Instead, the instructor, in the classroom setting, works with the group to solve problems.

Practicing for muscle memory

Here is a quote from the book, Art & Fear:

The ceramics teacher announced on opening day that he was dividing the class into two groups. All those on the left side of the studio, he said, would be graded solely on the quantity of work they produced, all those on the right solely on its quality. His procedure was simple: on the final day of class he would bring in his bathroom scales and weigh the work of the “quantity” group: fifty pound of pots rated an “A”, forty pounds a “B”, and so on. Those being graded on “quality”, however, needed to produce only one pot -albeit a perfect one – to get an “A”. Well, came grading time and a curious fact emerged: the works of highest quality were all produced by the group being graded for quantity. It seems that while the “quantity” group was busily churning out piles of work – and learning from their mistakes – the “quality” group had sat theorizing about perfection, and in the end had little more to show for their efforts than grandiose theories and a pile of dead clay.

If you want to learn JavaScript, write lot of code. Through writing the code, you keep improving. None of those “Learn X in 10 easy steps” can get you to this level of deep learning.

Hacking community realizes this aspect of learning well. Look at the following approach to learning by coding a lot: LxTHW (Learn X the hard way). The approach started with a popular book on Python, called Learn Python the Hard Way. If you follow that book, you don’t endlessly learn syntax and nuances of the language. Instead, you will code, and you will code a lot. It starts out this way:

  1. Do not cut and paste. Type the code as is.
  2. Make the code run.
  3. Now, modify the code to solve slightly different problems. And, make that code run.
  4. Keep repeating till you internalize all the concepts through practice.

In fact, this kind of learning through practice is applied by several people successfully. An example that I found very impressive was the case of Jennifer Dewalt. On April 1st, 2013, she started on a project to develop 180 websites in 180 days – one each per day. All these websites are simple enough to be coded in a day. With practice she got better and better; you can see the progress of her websites for yourself.

Even more experienced programmers like the inventor of JQuery, John Resig, feels that writing code everyday helps him keep his skils. Here is the famous blog post that he wrote: http://ejohn.org/blog/write-code-every-day/

In summary, our courses should not be teaching tricks and nuances of languages, libraries, or tools; they should be teaching the people practicing the craft.

Attention to basics

The big obstacle is coding or practicing the craft is not having the right basics. Even when two people are practicing equally, the one with the better basic tools will win.


Unfortunately, most colleges do not teach the tools of the trade. They focus, rightly on the fundamentals. Yet, there is no part of the education that covers the good use of tools and ways of working.

If practicing is the way to teach, the students need to have the right infrastructure to practice. In fact, practicing on that kind of infrastructure teaches them on how to use such infrastructure. These days, the basic infrastructure should be:

  1. Making use of the cloud
  2. Making use of the right OS, editors, and IDE
  3. Making use of the right tools for version control, bug tracking, requirements etc

Even for small projects, it is best to have at least a cloud based setup to try the code, a version control to keep the code history, and the right editor to train your muscles to use the right key strokes.

Quality through osmosis

We can get people to practice on the right infrastructure and churn out finished goods fast. But, will the output reach world-class quality?

While we made the argument that quality comes from quantity (through long practice), a more important ingredient is having right role models. That is where right mentorship can help. This is where intervention by a teacher that give feedback on quality of the output can help.

There are multiple ways we can bring this to practice – say if you are learning programming in JavaScript:


Especially, in the third step, and to lesser extent other two steps, good mentors can help. They can point out what the best practices are, idioms are, and why some practice is good.

Assessment through real-life tests

The current crop of testing, because of automation, is focused on multiple choice questions. Unfortunately, it only focuses on the nuances of the programming language or system. The world is far more complex; there is no single correct answer. Even if your problems were to come in the form of these questions, you could always find out answers from the internet.

In contrast, in real life, you will have to produce code or an artifact. Why not prepare you for that, during training? Here is an approach that works:

  1. Pick a large enough problem that we encounter during our work. Lot of these problems require “programming in the large”.
  2. Abstract it sufficiently so that it can be solved in limited time. For instance, you can choose only one module (say, adding new users to the system).
  3. Write a full fledged system that solves that sub-problem.
  4. If multiple people choose different parts of the system, then, we can have a fully functioning system at some point.

Training process

If you were to subscribe to this approach of corporate training, how would you start? Here is the approach I suggest.

  1. Start with a clear statement of what you want the students to be able to do.
    This part is surprisingly difficult. For instance, do not say “I want my students to learn Java”. That does not give them a new capability. Instead say “I want them to solve the following kinds of problems”. Now, these problems can be your way of assessing their status after the training.
  2. Create a pre-conditions for the material: Not just only for assessment, but as a way of setting expectations, you could ask the participants to do some tasks. For instance, if they are going to be doing JavaScript programming, they should know how to use Firefox or Chrome extensions. You could test for that.
  3. Create a curated content of the material: Since there is so much of material online, create an annotated list of material. For instance, you could give a link to the articles, books, slideshare entries or Youtube videos, with some notes about what the expectations from that videos are. You could construct them as a series of lessons.
  4. For each lesson, create a set of problems they should solve: In fact, more problems they practice, the better it is. If there is an entity like LxTHW, we could just follow that. If not, create a set of problems for them to solve so that the lessons really sink in.
  5. Create a basic infrastructure with tools: Even before they start solving problems, provide them an infrastructure such that:
    1. They can use the tools to develop the solutions
    2. They can use the infrastructure to collaborate with others
    3. They can use it to show the mentors
    4. They can test their solutions
  6. Provide the mentors: This part is tough. Just as we mentioned earlier, at the minimum, show them (embed in the curated content), what you consider as good quality to achieve.
  7. Create a post-training evaluation: Crate a large enough problem for the people to choose a part of the problem to solve. Using the mentors see
    1. how fast they are developing the solution (it indicates how they internalized the solution – practice makes for speed).
    2. how good a solution they are developing (it indicates how well they learnt from the masters)
  8. Create a community so that they can interact with each other post training: A training is not complete unless it is followed up. Since resources are difficult to get, use the power of community. Create a social space where people can discuss what they learnt even after they graduated from the course.

Concluding Remarks


I am no fan of corporate training; but I realize that not all people are self-motivated to learn and learn well. I think corporate training can be reformed to take advantage of the modern best practices such as incorporating internet material, using repetitive training, and intentional techniques in training, especially for acquiring technical capabilities. This note is an attempt towards that direction.

 Posted by at 1:12 pm