animesh kumar

Running water never grows stale. Keep flowing!

Archive for the ‘Uncategorized’ Category

WebSocket and node.js: why shud’ya care?

with 26 comments

[tweetmeme source=”anismiles” only_single=false http://www.URL.com%5D

Traditional HTTP messages are heavy. Every message is sent with HTTP headers. Now, let’s say you have an application that has a real-time component, like chat or some twitter client or may be some traffic analysis stuff. And let’s say you have around 100,000 users connected to your app. To make your app real-time, you need to have a mechanism which will enable server to push data almost as soon as this data becomes available. You could do it in two ways: Write a script which will connect to server every few seconds to check if there is any data. With each attempt, full set of HTTP headers moves back and forth between client and server. That’s not very efficient. To save yourself with all these bandwidth hassles, you could use a popular trick known as long-polling, where your browser connects to server and server holds the connection open until there is some data available to be pushed.

Now, let’s assume that there are 100,000 users connected to your app and every 10 seconds some data is sent from server to clients. Following HTTP specs, every time some data is sent, full set of headers are shared between client and server. This is how they look,

Request header

GET / HTTP/1.1
User-Agent: ...some long user agent string...
Host: animesh.org
Accept: */*

Response header

HTTP/1.1 200 OK
Date: Tue, 25 Jan 2011 17:32:19 GMT
Server: Apache
X-Powered-By: PHP/5.2.3
X-Pingback: http://animesh.org/endpoint/
Connection: close
Transfer-Encoding: chunked
Content-Type: text/html; charset=UTF-8

That’s approximately 350 bytes of data, per user every 10 seconds. That’s roughly 28,400,000 bits per second of network throughput for 100,000 users. Roughly 26.7 Mbps for only HTTP headers. Gosh!

WebSocket

WebSocket comes to resue. With web sockets, once a handshake is done between client and server, messages can be sent back and forth with a minimal overhead. That’s awesome. You do a handshake while establishing the connection, and of course handshaking needs all those HTTP headers, but after that, you only need to send the data… no headers. This greatly reduces the bandwidth usage and thus improves the performance. Let’s see how. This is how handshake headers look like,

Handshake Request header

GET /demo HTTP/1.1
Upgrade: WebSocket
Connection: Upgrade
Host: animesh.org
Origin: http://animesh.org
WebSocket-Protocol: sample

Handshake Response header

HTTP/1.1 101 Web Socket Protocol Handshake
Upgrade: WebSocket
Connection: Upgrade
WebSocket-Origin: http://animesh.org
WebSocket-Location: ws://animesh.org/
WebSocket-Protocol: sample

And now, the connection has been established and data can freely flow between server and client without having to exchange any HTTP headers until this connection is closed or broken and you do another handshake. Imagine how much bandwidth you are saving! Whoa!

Example

Let’s write a simple application to see and learn how this thing actually works. This application will have a server all the clients will connect to, and whenever one client writes something to the server, all clients will be notified.

Here is our server, written in Node.js. Let’s name it server.js

Note: Though you can very well write a web socket server using Node’s native APIs, however I chose to use Micheil Smith‘s node-websocket-server library. This library is simple, elegant and very easy to work with. It works by wrapping and extending Node’s server object.

var sys = require("sys");
// Library https://github.com/miksago/node-websocket-server
var	websocket = require('./lib/node-websocket-server/lib/ws/server');

// create web socket server
var server = websocket.createServer();
// listen on port 8078
server.listen(8078);

// when the server is ready
server.addListener("listening", function() {
  sys.log("Listening for connections on localhost:8078");
});

// when a traditional HTTP request comes
server.addListener("request", function(req, res) {
	res.writeHead(200, {
		"Content-Type" : "text/plain"
	});
	res.write("This is an example WebSocket server.");
	res.end();
});

// when a client websocket connects
server.addListener("connection", function(conn) {

	// when client writes something
	conn.addListener("message", function(message) {

		// iterate thorough all connected clients, and push this message
		server.manager.forEach(function(connected_client) {
			connected_client.write(JSON.stringify(conn.id + ": " + message));
        });
	});
});

Now, let’s write a simple client. We will create one HTML file and run it in Google Chrome. Let’s name is client.html

<!DOCTYPE html>
<html>
<head>
    <title>WebSocket - Simple Client</title>
    <script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.2.6/jquery.js"></script>
    <script type="text/javascript">

	$(function() {
		// bind form
		$('#payload-form').submit(function() {
			var payload = ($("input#payload").val());
			socket.send(payload);  // write to server
			return false;
		});

		// open websocket
		var socket = new WebSocket('ws://localhost:8078');

		socket.onopen = function() {
	    	// Web Socket is connected. send an initial random message.
	    	socket.send(Math.floor(Math.random()*11) + ' >> Hi, I am Mr. so-and-so!');
	    };
		// append to '#log' whatever server pushes.
		socket.onmessage = function(ev){
			msg = JSON.parse(ev.data);
			$('#log').append(JSON.stringify(msg) + '</br>');
		}
	})
    </script>
</head>
<body>
	<div id='payload-container'>
		<form id='payload-form'>
			<input type='text' name='payload' id='payload' value='Hello World' style="width:500px;"/>
			<input type='submit' value='push'/>
		</form>
	</div>

	<div id='log' style='display:block; border:1px solid lightgray;'></div>
</body>
</html>

Now, run your server, and open your client in multiple Chrome windows/tabs.

// run server
$ node server.js

That’s it! Was is fun? I will write more on how to establish WebSocket connections from a Java client in the next blog.

Written by Animesh

January 25, 2011 at 3:25 pm

Maven, SQL and ordered execution

leave a comment »

[tweetmeme source=”anismiles” only_single=false http://www.URL.com%5D

I have been re-architecting this new project. This is huge. And broken. And I was called upon to clean, refactor and re-architect it. Phew!

I thought a good strategy would be to deal with it in outside-in style, which is, fixing the build part first then moving down to various parts. One of the interesting things that they had done was using Maven to populate PostgreSQL DB Schemas and then load data into the DB.

Steps of execution

  1. Drop everything
  2. Create users
  3. Create schemas
  4. Load data
  5. Load upgrade data

They used sql-maven-plugin underwhich they had multiple execution tasks. The whole thing was very untidy and I didn’t like it.

Summary

  1. They used <fileSets> to import sql files. FileSets are an elegant choice if you want to import files with ANT like wildcard patterns. But the files are alphabetically sorted. So, unless you don’t care about the order in which the files will be executed, you should avoid using FileSets. In their case, they wanted to run ‘drop_users_databases.sql’ before ‘create_users_databases.sql’. With FileSets, it was impossible to do, they would be re-ordered just the opposite. Solution? They should have used <srcFiles>, but instead they created 2 separate <execution> tasks to run in order. Too much verbosity. Eh?
  2. Another issue was with loading upgrade data. Now, they follow agile methodologies and keep updating their DB schemas, and so after few iterations, before they merge the changes in the main script, they end up with a number of files like upgrade-schema-2.0.4.sql, upgrade-schema-2.0.5.sql, upgrade-schema-2.0.5.1.sql and so on. Pain is, they don’t have any automated mechanism to execute these files in order, so the developers would be forced to do it manually. If the number of these files were low, that wouldn’t have been a problem… but how about thirty or forty such files? Pissed off.

Strategy

  1. Kick <fileSets> out of the window
  2. Use <srcFiles> instead (because I cared more about order)
  3. Logically partition the operations
  4. Create a new profile to help run these DB executions at will

Code

<profiles>  
<profile>  
	<!-- profile id -->
	<id>init-db</id> 
	<build>
		<plugins>
			<plugin>
				<groupId>org.codehaus.mojo</groupId>
				<artifactId>sql-maven-plugin</artifactId>
				
				<!-- postgresql dependencies -->	
				<dependencies>
				  <dependency>
					<groupId>postgresql</groupId>
					<artifactId>postgresql</artifactId>
					<version>8.3-603.jdbc4</version>
					<scope>clean</scope>
				  </dependency>
				</dependencies>
				
				<!-- DB Configuration -->
				<configuration>
					<url>jdbc:postgresql:my_db</url>
					<driver>org.postgresql.Driver</driver>
					<username>user</username>
					<password>pass</password>
					<autocommit>true</autocommit>							
				</configuration>
				
				<executions>
					<!-- 1. drop everything, create users and load data -->
					<execution>
						<id>drop-and-create-database</id>
						<phase>clean</phase>
						<goals><goal>execute</goal></goals>
						<configuration>
							<srcFiles>
								<srcFile>src/main/sql/drop_users_databases.sql</srcFile>						
								<srcFile>src/main/sql/create_users_databases.sql</srcFile>						
								<srcFile>src/main/sql/create_schema.sql</srcFile>						
								<srcFile>src/main/sql/load_data.sql</srcFile>						
								<srcFile>src/main/sql/test_data.sql</srcFile>						
							</srcFiles>    															
						</configuration>
					</execution>			
					<!-- 2. run all upgrade scripts -->
					<execution>
						<id>upgrade-schema</id>
						<phase>clean</phase>
						<goals><goal>execute</goal></goals>
						<configuration>
							<srcFiles>
								<srcFile>src/main/sql/upgrade-schema-2.0.6.sql</srcFile>						
								<srcFile>src/main/sql/upgrade-schema-2.1.5.sql</srcFile>						
								<srcFile>src/main/sql/upgrade-schema-2.1.7.sql</srcFile>						
								<srcFile>src/main/sql/upgrade-schema-2.1.8.sql</srcFile>
								<srcFile>src/main/sql/upgrade-schema-2.2.1.sql</srcFile>
								<srcFile>src/main/sql/upgrade-schema-3.0.0.sql</srcFile>
								<srcFile>src/main/sql/upgrade-schema-3.0.1.sql</srcFile>
								<srcFile>src/main/sql/upgrade-schema-3.0.2.sql</srcFile>
								<srcFile>src/main/sql/upgrade-schema-3.0.3.sql</srcFile>
								<srcFile>src/main/sql/upgrade-schema-3.0.4.sql</srcFile>
								<srcFile>src/main/sql/upgrade-schema-3.0.5.sql</srcFile>
								<srcFile>src/main/sql/upgrade-schema-3.0.6.sql</srcFile>
								<srcFile>src/main/sql/upgrade-schema-3.0.6.1.sql</srcFile>
								<srcFile>src/main/sql/upgrade-schema-3.0.8.sql</srcFile>
								<srcFile>src/main/sql/upgrade-schema-3.0.9.sql</srcFile>
								<srcFile>src/main/sql/upgrade-schema-3.0.10.sql</srcFile>
								<srcFile>src/main/sql/upgrade-schema-3.0.11.sql</srcFile>
							</srcFiles>    						
						</configuration>
					</execution>
				</executions> 
			</plugin>
		</plugins>  
	</build>  
</profile>  
</profiles>  	  

How to run?

Note that I have bound the executions with phase ‘clean’ within a profile with id ‘init-db’. So, I just need to let maven know about the phase and profile, like this:

mvn -Pinit-db clean

I hope this will help someone in need. I will post more adventurous code cleaning stuffs as I will encounter them. Cheers!

Written by Animesh

December 28, 2010 at 12:31 pm

wooing consumers (when Selling is about Killing)

leave a comment »

How can you explain “two for the price of one”, “easy monthly payments”, “money back guarantee” gimmicks? Marketers have been using these tricks and there is no scientific proof why people prefer free incentives when in the back of their minds they know that there is nothing free, they will have to pay for it, this way or that.

Consumerism is irrational. Consumer is irrational, illogical and that’s how he thinks about his money. He applies different “mental models” for money coming from different sources, for example, he finds it easiest to spend pocket money, harder to spend his income, and hardest to relinquish his savings. Every rupee is different. Now, isn’t that irrational? Counter intuitive? Yes, but that’s how it is.

But no matter what, money going off his hands is always painful.

  1. Make a less-painful product: Choosing not to choose is in itself a choice.
    A consumer can always defer to purchase, saving money for another day. And that’s a problem, worse that your competition. So, allow delay payments, or easy installments. A mitigated way for consumer to part with his money. Incorporate window shopping. Let customers have a feel of the product, generate emotion for it, feel attached and then worry about the money.

  2. Assert your offer, sell by default. Idea is to make your consumers feel the ownership before you ask them to make a decision. If you have a service to sell, don’t ask him to sign up for it. Instead, sign him up by default, and then ask him whether he wants to cancel it? For a telecom company, it would be better to ask, “We have already credited your account with 100 calls, how do you want to use them?” A consumer would not like to let go of calls he already owns. You will have more acceptance rate.
  3. Limit his choices: In a classic field experiment, some grocery store shoppers were offered the chance to taste a selection of 24 jams, while others were offered only 6. The greater variety drew more shoppers to sample the jams, but few made a purchase. By contrast, although fewer consumers stopped to taste the 6 jams on offer, sales from this group were more than five times higher.
  4. Comparatively placement: Many restaurants find that the second-most-expensive bottle of wine is very popular—and so is the second-cheapest. Customers who buy the former feel they are getting something special but not going over the top. Those who buy the latter feel they are getting a bargain but not being cheap. The power of relative positioning explains why marketers sometimes benefit from offering a few clearly inferior options. Even if they don’t sell, they may increase sales of slightly better products the store really wants to move.

Any thoughts?

Read more here:  Mckinsey Quarterly

Written by Animesh

February 25, 2010 at 2:40 am

Posted in Uncategorized