animesh kumar

Running water never grows stale. Keep flowing!

Jesque-Guice binding

with one comment

In our Java stack, we use Guice IOC library quite heavily. We are kind of smitten with it. Now, when we needed a task queue to process our background jobs, we settled with Jesque which is a port of Github’s Resque project, and runs on Redis which was already part of our stack. In the process, we also added delayed task functionality to this.

Problem

Anyways, the problem we were facing was with Injecting Guice dependencies into Jesque workers. Our workers performed heavy operations and at times had to read-write from DB, or speak with other services.

Jesque uses Java reflection APIs to instantiate and run its workers (which are Runnable classes) and so, it becomes very difficult to inject Guice managed objects and services into Jesque workers.

We started the hacky-and-ugly way by creating static references to relevant services that our workers needed. if it were for few services, this would have worked, but our workers kept growing in features and reach, and after a while the whole code base was stinking. We had to do something about it. Something elegant!

Solution

So, we created Jesque Guice binding project. You can annotate your worker classes, and Guice will then discover, register and start them. Let me show you some code.

First, let’s create an ExampleWorker. You have to remember that the Worker must implement Runnable interface, and it must be annotated with @Worker which Guice uses for discovery and binding.

// ExampleWorker

@Worker(job = "ExampleJob",                 // job name
        queues = { "EXP_QUEUE" },           // queue names
        enabled = true,                     // enabled
        count = 1,                          // 1 instance of this worker running
        events = { WorkerEvent.JOB_SUCCESS, WorkerEvent.WORKER_START }, // Events to listen to
        listener = EchoListener.class      // WorkerEventListener
)
public class ExampleWorker implements Runnable {
    // LOG
    private static final Logger LOG = LoggerFactory.getLogger(ExampleWorker.class);

    // Note: Only field level injection would work!
    @Inject
    ExampleService service;

    String arg1;
    String arg2;

    // Must keep an empty constructor for Guice to discover this
    public ExampleWorker() {
    }

    public ExampleWorker(String arg1, String arg2) {
        this.arg1 = arg1;
        this.arg2 = arg2;
    }

    @Override
    public void run() {
        LOG.info("Running worker={}, with arg1={}, arg2={}", new Object[] { getClass(), arg1, arg2 });
        // calling service
        service.serve();
    }

}

@Worker attributes let you control the behavior. The above worker, ExampleWorker, listens to EXP_QUEUE queue, accepts Jobs by name ExampleJob, has an WorkerEventListener defined by Guice Managed class EchoListener which listens to WorkerEvent JOB_SUCCESS and WORKER_START. There is only ONE instance of this worker running.

Note: ExampleWorker has been field @Injected with ExampleService. Please remember, that
Only field @Inject will work with Workers, because Jesque uses constructors to pass Job arguments.
– Also, you must keep an empty constructor around so as to let Guice discover this worker.

Now, since we have added a Listener, we must define it.

// EchoListener

public class EchoListener implements WorkerListener {
    // LOG
    private static final Logger LOG = LoggerFactory.getLogger(EchoListener.class);

    @Override
    public void onEvent(WorkerEvent event, Worker worker, String queue, Job job, Object runner,
            Object result, Exception ex) {
        LOG.info("onEvent ==>> queue={}, event={}", new Object[] { queue, event });
    }

}

For the sake of demonstration, this has been kept very minimal. But, mind you, you can @Inject any Guice managed objects into this, using constructor and/or field injection.

Now, let’s define the ExampleService that we want to @Inject into our worker.

// ExampleService

public class ExampleService {
    /** The Constant LOG. */
    private static final Logger LOG = LoggerFactory.getLogger(ExampleService.class);

    public void serve() {
        LOG.info("Heya! I am not here to serve, just to show you how injection works.");
    }
}

Wonderful! Let’s now bind these all together in a GuiceModule.

// ExampleModule

public class ExampleModule extends AbstractModule {

    @Override
    protected void configure() {
        // Jesque Guice
        install(new JesqueModule());

        // Jesque Client
        Config config = new ConfigBuilder().withHost("localhost").withPort(6379).withDatabase(0).build();
        bind(Config.class).toInstance(config);
        bind(Client.class).toInstance(new ClientImpl(config));

        // Worker
        bind(ExampleWorker.class).asEagerSingleton(); // Must be singleton
        // WorkerEventListener
        bind(EchoListener.class).in(Scopes.SINGLETON);
        // Worker Executor (This is where they actually run)
        bind(WorkerExecutor.class).to(SimpleThreadBasedWorkerExecutor.class);
        // Service (will be injected into workers)
        bind(ExampleService.class).asEagerSingleton();
    }

}

Here, first we install JesqueModule, and then bind other objects.
– Bind Jesque config and client.
– Bind Worker, Lister, Service etc.

You will notice we have also bound WorkerExecutor. This interface accepts net.greghaines.jesque.worker.Worker instance and runs that on a thread. Jesque-Guice comes with 2 simple implementations:

1. SimpleThreadBasedWorkerExecutor which run each net.greghaines.jesque.worker.Worker on an unmanaged separate thread, and
2. CachedThreadPoolBasedWorkerExecutor which creates a CachedThreadPool where net.greghaines.jesque.worker.Worker is run.

You can implement your own strategy or provide your own ExecutorService.

Run

Now we have everything, let’s run it then.

// Main

public class Main {
    /** The Constant LOG. */
    private static final Logger LOG = LoggerFactory.getLogger(Main.class);

    /**
     * @param args
     */
    public static void main(String[] args) {
        Injector injector = Guice.createInjector(Stage.DEVELOPMENT, new Module[] { new ExampleModule() });

        // Get Jesque client
        Client client = (Client) injector.getInstance(Client.class);
        LOG.info("Publish jobs");
        // Push jobs
        client.enqueue("EXP_QUEUE", new Job("ExampleJob", "hello", "job1"));
        client.enqueue("EXP_QUEUE", new Job("ExampleJob", "hello", "job2"));
    }
}

You should see,

// Job - 1
DEBUG c.s.commons.jesque.GuiceAwareWorker - Injecting dependencies into worker instance = com.strumsoft.commons.jesque.example.ExampleWorker@6b754699
DEBUG c.s.commons.jesque.GuiceAwareWorker - Delegating to run worker instance = com.strumsoft.commons.jesque.example.ExampleWorker@6b754699
INFO  c.s.c.jesque.example.ExampleWorker - Running worker=class com.strumsoft.commons.jesque.example.ExampleWorker, with arg1=hello, arg2=job1
INFO  c.s.c.jesque.example.ExampleService - Heya! I am not here to serve, just to show you how injection works.
INFO  c.s.c.jesque.example.EchoListener - onEvent ==>> queue=EXP_QUEUE, event=JOB_SUCCESS

// Job -2
DEBUG c.s.commons.jesque.GuiceAwareWorker - Injecting dependencies into worker instance = com.strumsoft.commons.jesque.example.ExampleWorker@6602e323
DEBUG c.s.commons.jesque.GuiceAwareWorker - Delegating to run worker instance = com.strumsoft.commons.jesque.example.ExampleWorker@6602e323
INFO  c.s.c.jesque.example.ExampleWorker - Running worker=class com.strumsoft.commons.jesque.example.ExampleWorker, with arg1=hello, arg2=job2
INFO  c.s.c.jesque.example.ExampleService - Heya! I am not here to serve, just to show you how injection works.
INFO  c.s.c.jesque.example.EchoListener - onEvent ==>> queue=EXP_QUEUE, event=JOB_SUCCESS

Works… eh? :)

You can grab the project source at github https://github.com/anismiles/jesque-guice Give it a try. I hope this will help some of you folks. Share your thoughts with me.

Happy hacking!

Written by Animesh

September 18, 2013 at 4:44 pm

Delayed Jobs with Jesque

with 3 comments

Jesque is an interesting project. It’s a Java port of Github’s Resque task queue library. Works on top of Redis. It is fast. It eats less. And doesn’t need a lot of maintenance, as long long your Redis is okay, you are okay! We have been happily using this in production for a while.

Problem

And then, we encountered a need for delayed jobs. We needed an ability to execute jobs in future with deterministic delay. We had 2 options:

1. Either, schedule these delayed jobs in Quartz – that was okay because we already had been using Quartz – and then, once Quartz jobs get fired, they publish a Task into Jesque, and let the workers handle the rest. (This was too contrived to implement, and would have become maintenance/reporting nightmare!)

2. We extend Jesque to support delayed jobs inherently.

Solution

We decided to go with option-2, and started exploring Redis datasets. Turned out, ZSET was all that we needed.

Jesque uses Redis LIST for job storage, workers keep polling the list, and LPOPs tasks from the LIST.

This is what we ended up doing:

When adding a delayed job,

1. Calculate future timestamp when the job should run,
2. Use that timestamp as SCORE to ZSET entry.

// Java
final long delay = 10; //sec
final long future = System.currentTimeMillis() + (delay * 1000); // future
jedis.zadd(QUEUE, future, jobInfo); 

// Redis
// ZADD <queue> <future> <job-information>

On the other hand, Workers’ poll logic was updated. For Delayed queues,

1. Check if there are any items with SCORE between -INF and now,

final long now = System.currentTimeMillis();
final Set<String> tasks = jedis.zrangeByScore(QUEUE, -1, now, 0, 1);

If tasks are non-empty, try to grab one to execute.

if (null != tasks && !tasks.isEmpty()) {
    String task = tasks.iterator().next();
    // try to acquire this task
    if (jedis.zrem(QUEUE, task) == 1) {
         return task; // Return
    }
}

This way, we ensure that no 2 workers would grab the same task to execute.

Also, an important point to note here is that – You don’t have to change your existing workers or redo new workers in any way. Just bind them to a Delayed Queue, and start publishing delayed tasks.

Example

// DelayedJobTest.java
package net.greghaines.jesque;

import static net.greghaines.jesque.utils.JesqueUtils.entry;
import static net.greghaines.jesque.utils.JesqueUtils.map;

import java.util.Arrays;

import net.greghaines.jesque.client.Client;
import net.greghaines.jesque.client.ClientPoolImpl;
import net.greghaines.jesque.worker.Worker;
import net.greghaines.jesque.worker.WorkerImpl;
import redis.clients.jedis.JedisPool;

public class DelayedJobTest {

    @SuppressWarnings("unchecked")
    public static void main(String[] args) throws InterruptedException {
        // Queue name
        final String QUEUE = "fooDelayed";
        
        // Config
        final Config config = new ConfigBuilder().withHost("localhost").withPort(6379).withDatabase(0).build();
        
        // Client
        final Client client = new ClientPoolImpl(config, new JedisPool("localhost"));
        long delay = 10; // seconds
        long future = System.currentTimeMillis() + (delay * 1000); // Future timestamp
        
        // Enqueue job
        client.delayedEnqueue(QUEUE, 
                new Job(
                        TestJob.class.getSimpleName(), 
                        new Object[] {"HELLO", "WORLD" } // arguments
                ), 
                future);
        // End
        client.end();

        // Worker
        final Worker worker = new WorkerImpl(config, Arrays.asList(QUEUE), map(entry(TestJob.class.getSimpleName(), TestJob.class)));
        final Thread workerThread = new Thread(worker);
        workerThread.start(); // start
    }
}

And, this is the TestJob,

// TestJob.java
package net.greghaines.jesque;

import java.util.Date;

public class TestJob implements Runnable {
    final String arg1;
    final String arg2;

    public TestJob(String arg1, String arg2) {
        this.arg1 = arg1;
        this.arg2 = arg2;
    }

    @Override
    public void run() {
        System.out.println("Job ran at=" + new Date() + ", with arg1=" + arg1 + " and arg2=" + arg2);
    }
}

We have using this solution in production for quite sometime, and this has been pretty stable.

BTW, you can grab the source here: https://github.com/anismiles/jesque and give it a try yourself.

Happy Hacking!

Written by Animesh

September 2, 2013 at 5:06 pm

Jetty Executable Webserver Archetype

with 2 comments

We create and deploy many rest based API apps on our servers. Now, instead of managing multitudes of webservers on different machines, we make our deployable archives wrapped with all the necessary dependencies in order to be self-executable. Benefits?

  • We just have to ensure that all our servers have the same Java environment, and no other dependencies.
  • No waste of time configuring, managing web (tomcat, jetty etc.) servers.
  • And, we only have to distribute a single executable.

Hassle free management. Won’t you agree?

To cut down on bootstrapping time, we have created a maven archetype to help us quickly create and start on new API projects. Our tech stack is Google Guice, JBoss Resteasy, Logback, and Embedded Jetty.

Github link: https://github.com/anismiles/jetty-webserver-archetype

Install

You can either build/install from source

git clone https://github.com/anismiles/jetty-webserver-archetype.git
mvn clean install

Or you can simply download the jar and install that.

mvn install:install-file \
-Dfile=jetty-webserver-archetype-1.0.jar \
-DgroupId=com.strumsoft \
-DartifactId=jetty-webserver-archetype \
-Dversion=1.0 \
-Dpackaging=jar \
-DgeneratePom=true

Usage

Let’s say you want to create a new project “hello-world” with group “com.hello.world”, you would run:

mvn archetype:generate \
-DgroupId=com.hello.world \
-DartifactId=hello-world \
-Dversion=1.0-SNAPSHOT \
-DarchetypeGroupId=com.strumsoft \
-DarchetypeVersion=1.0 \
-DarchetypeArtifactId=jetty-webserver-archetype

That’s it. You are ready to roll. You have set up a basic working API app. Now, you can run it in dev mode

mvn clean jetty:run

Or in production mode

mvn clean package
java –jar target/hello-world-1.0-SNAPSHOT-dist.war start &

To check, open http://localhost:8085/
To stop:

java –jar target/hello-world-1.0-SNAPSHOT-dist.war stop

Further, you can pass additional Jetty, Logback or your App’s properties

java \
-Djetty.configurationFile=<jetty-config> \
-Dapp.configurationFile=<app.properties> \
-Dlogback.configurationFile=<logback.xml> \
–jar target/hello-world-1.0-SNAPSHOT-dist.war start &

They will override default properties setup by the executable.
Ideally you might want to create an init.d file to start and stop you API app efficiently. Here is a prototype:

#!/bin/sh

# Executable war file
WAR_FILE=/var/apps/<my-war-file>.war

# Configuration files
APP_NAME=<app-name>

JETTY_CONFIG=/etc/${APP_NAME}-jetty.properties
APP_CONFIG=/etc/${APP_NAME}-app.properties
LOGBACK_CONFIG=/etc/${APP_NAME}-logback.properties

# Process
PIDFILE=/var/run/${APP_NAME}.pid

# Java arguments
JAVA_ARGS="-Djetty.configurationFile=${JETTY_CONFIG} -Dapp.configurationFile=${APP_CONFIG} -Dlogback.configurationFile=${LOGBACK_CONFIG}"

# Command
CMD=$1

if [[ -z ${CMD} ]]; then
  java ${JAVA_ARGS} -jar ${WAR_FILE} usage
  exit 1
fi

if [[ ${CMD} == 'start' ]]; then
    if [[ -f ${PIDFILE} ]]; then
        echo "Already running"
        exit 1
    fi

    if [[ -f ${WAR_FILE} ]]; then
        echo "Starting jetty: ${WAR_FILE}"
        java ${JAVA_ARGS} -jar ${WAR_FILE} start &
        PID=$!
        echo "$PID" > ${PIDFILE}
        echo "Started ${APP_NAME} with pid: ${PID}"
    fi

elif [[ ${CMD} == 'stop' ]]; then
    # Try gracefully first
    java ${JAVA_ARGS} -jar ${WAR_FILE} stop
    sleep 10
    if [[ -f ${PIDFILE} ]]; then
        PID=`cat ${PIDFILE}`
        test -z $PID || kill $PID
        rm ${PIDFILE}
        sleep 1
        echo "Forcibly Stopped ${WAR_FILE} with pid: ${PID}"
    fi

else # Just let the other cmds through...
    java ${JAVA_ARGS} -jar ${WAR_FILE} ${CMD}
fi

exit 0

That’s it! have fun. :)

Written by Animesh

January 8, 2013 at 1:48 pm

Securing, Versioning and Auditing REST (JAX-RS, Jersey) APIs

with 29 comments

Now that your functionalities are working, you want a layer of security to authenticate/authorize your APIs. Though this is a bad approach towards security, but – I know – real life is a tough game and nothing happens they way they should be… and so be it. Additionally, you might want to control API versions (i.e. expose newer APIs only to newer clients) and audit API usage.

Well, I am going to propose a tangential way to implement these concerns. You won’t need to touch any of your business logic as such. Only few annotations (custom and otherwise) would need to be applied. That way, you won’t feel bad about missing these things when you started the project and your concerns will be taken care of in the most un-obtrusive way possible. Wonderful… eh?

First, you will need to create some sort of sign-in API, which will accept username/password (or oAuth or whatever you fancy) and generate some sort of session information which you will store in some database (Redis maybe!) and share its ID, say sessionId, with client. Then, with every subsequent request, Client will attach this sessionId in the request header which server will pick and look up for associated session information (permission, roles etc.) and based upon that server will authenticate and authorize the request.

Here is your Session bean.

package com.strumsoft.api;

import java.io.Serializable;
import java.util.Date;

/**
 * @author "Animesh Kumar <animesh@strumsoft.com>"
 *
 */
public class Session implements Serializable {
	// 
	private static final long serialVersionUID = -7483170872690892182L;
	
	private String sessionId;   // id
	private String userId;      // user
	private boolean active;     // session active?
	private boolean secure;     // session secure?

	private Date createTime;    // session create time
	private Date lastAccessedTime;  // session last use time

	// getters/setters here
}

And this is your User bean. This must implement java.security.Principal.


package com.strumsoft.api;

import java.util.Set;

/**
 * @author "Animesh Kumar <animesh@strumsoft.com>"
 *
 */
public class User implements java.security.Principal {
	// Role
	public enum Role {
		Editor, Visitor, Contributor
	};

	private String userId;          // id
	private String name;            // name
	private String emailAddress;    // email
	private Set<Role> roles;        // roles

	@Override
	public String getName() {
		return null;
	}

	// getters/setters here
}

Now, you need to implement javax.ws.rs.core.SecurityContext. This will be bound to incoming request and will decide whether to allow or deny it.

package com.strumsoft.api;

import java.security.Principal;
import javax.ws.rs.WebApplicationException;
import javax.ws.rs.core.Response;
import javax.ws.rs.core.SecurityContext;

/**
 * @author "Animesh Kumar <animesh@strumsoft.com>"
 *
 */
public class MySecurityContext implements javax.ws.rs.core.SecurityContext {

	private final User user;
	private final Session session;

	public MySecurityContext(Session session, User user) {
		this.session = session;
		this.user = user;
	}

	@Override
	public String getAuthenticationScheme() {
		return SecurityContext.BASIC_AUTH;
	}

	@Override
	public Principal getUserPrincipal() {
		return user;
	}

	@Override
	public boolean isSecure() {
		return (null != session) ? session.isSecure() : false;
	}

	@Override
	public boolean isUserInRole(String role) {

		if (null == session || !session.isActive()) {
			// Forbidden
			Response denied = Response.status(Response.Status.FORBIDDEN).entity("Permission Denied").build();
			throw new WebApplicationException(denied);
		}

		try {
			// this user has this role?
			return user.getRoles().contains(User.Role.valueOf(role));
		} catch (Exception e) {
		}
		
		return false;
	}
}

Then, you need a ResourceFilter which will intercept the request, look for sessionId in the header and generate and attach our SecurityContext implementation to it. Notice, how our implementation only gets applied on Request but not on Response.

package com.strumsoft.api;

import javax.ws.rs.ext.Provider;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;

import com.flockthere.api.repository.SessionRepository;
import com.flockthere.api.repository.UserRepository;
import com.sun.jersey.spi.container.ContainerRequest;
import com.sun.jersey.spi.container.ContainerRequestFilter;
import com.sun.jersey.spi.container.ContainerResponseFilter;
import com.sun.jersey.spi.container.ResourceFilter;

/**
 * Filter all incoming requests, look for possible session information and use that
 * to create and load a SecurityContext to request. 
 * @author "Animesh Kumar <animesh@strumsoft.com>"
 * 
 */
@Component   // let spring manage the lifecycle
@Provider    // register as jersey's provider
public class SecurityContextFilter implements ResourceFilter, ContainerRequestFilter {

	@Autowired
	private SessionRepository sessionRepository;  // DAO to access Session

	@Autowired
	private UserRepository userRepository;  // DAO to access User

	
	@Override
	public ContainerRequest filter(ContainerRequest request) {
		// Get session id from request header
		final String sessionId = request.getHeaderValue("session-id");

		User user = null;
		Session session = null;

		if (sessionId != null && sessionId.length() > 0) {
			// Load session object from repository
			session = sessionRepository.findOne(sessionId);
			
			// Load associated user from session
			if (null != session) {
				user = userRepository.findOne(session.getUserId());
			}
		}

		// Set security context
		request.setSecurityContext(new MySecurityContext(session, user));
		return request;
	}

	@Override
	public ContainerRequestFilter getRequestFilter() {
		return this;
	}

	@Override
	public ContainerResponseFilter getResponseFilter() {
		return null;
	}
}

Okay, the hard part is over. All we need now is a way to fire our SecurityContextFilter. For this, we will create a ResourceFilterFactory. During application startup, this factory will create a List of filters for all AbstractMethods of each of our Resources. We are going to extend RolesAllowedResourceFilterFactory which will generate all Role based ResourceFilters for us. And then, we will add our SecurityContextFilter on the top of the list with VersionFilter and AuditFilter in the bottom. That way, SecurityContextFilter will executed first because you need to make auth decisions early. VersionFilter will be next. And Audit in the bottom. You want to audit when everything else has been done. No?

package com.strumsoft.api;

import java.util.ArrayList;
import java.util.List;

import javax.ws.rs.ext.Provider;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;

import com.flockthere.api.AllowAllVersions;
import com.flockthere.api.audit.Audit;
import com.flockthere.api.resource.interceptor.AuditingFilter;
import com.flockthere.api.resource.interceptor.SecurityContextFilter;
import com.flockthere.api.resource.interceptor.VersionFilter;
import com.sun.jersey.api.container.filter.RolesAllowedResourceFilterFactory;
import com.sun.jersey.api.model.AbstractMethod;
import com.sun.jersey.spi.container.ResourceFilter;

/**
 * FilterFactory to create List of request/response filters to be applied on a particular
 * AbstractMethod of a resource.
 * 
 * @author "Animesh Kumar <animesh@strumsoft.com>"
 * 
 */
@Component // let spring manage the lifecycle
@Provider  // register as jersey's provider
public class ResourceFilterFactory extends RolesAllowedResourceFilterFactory {

	@Autowired
	private SecurityContextFilter securityContextFilter;

	// Similar to SecurityContextFilter to check incoming requests for API Version information and
	// act accordingly
	@Autowired
	private VersionFilter versionFilter;

	// Similar to SecurityContextFilter to audit incoming requests
	@Autowired
	private AuditingFilter auditingFilter;

	@Override
	public List<ResourceFilter> create(AbstractMethod am) {
		// get filters from RolesAllowedResourceFilterFactory Factory!
		List<ResourceFilter> rolesFilters = super.create(am);
		if (null == rolesFilters) {
			rolesFilters = new ArrayList<ResourceFilter>();
		}

		// Convert into mutable List, so as to add more filters that we need
		// (RolesAllowedResourceFilterFactory generates immutable list of filters)
		List<ResourceFilter> filters = new ArrayList<ResourceFilter>(rolesFilters);

		// Load SecurityContext first (this will load security context onto request)
		filters.add(0, securityContextFilter);

		// Version Control?
		filters.add(versionFilter);

		// If this abstract method is annotated with @Audit, we will apply AuditFilter to audit
		// this request.
		if (am.isAnnotationPresent(Audit.class)) {
			filters.add(auditingFilter);
		}

		return filters;
	}
}

VersionFilter will help control Client’s access to API methods based upon client’s version. Implementation would be similar to SecurityContextFilter. You will need to annotate API methods with @Version(“>1.3″). VersionFilter will read this value (“>1.3″), check request headers for API-Version keys and then decide whether to allow or reject the request. Similarly, AuditFilter will log all such annotated (@Audit(“audit-note”)) API methods. I will not discuss their actual implementations. You can very easily write them based upon your requirement or can remove them altogether if not needed.

Here is how these annotations will look like.

// @Version
@Retention(RetentionPolicy.RUNTIME)
@Target({ ElementType.METHOD })
public @interface Version {
	String value();
}

// @Audit
@Retention(RetentionPolicy.RUNTIME)
@Target({ ElementType.METHOD })
public @interface Audit {
	String value();
}

And now, you will need to update you web.xml to include ResourceFilterFactory

...
	<servlet>
		<servlet-name>REST</servlet-name>
		<servlet-class>
			com.sun.jersey.spi.spring.container.servlet.SpringServlet
		</servlet-class>
		...		
		<!-- Our resource filter for tangential concerns (security, logging, version etc.) -->
		<init-param>
			<param-name>com.sun.jersey.spi.container.ResourceFilters</param-name>
			<param-value>com.strumsoft.api.ResourceFilterFactory</param-value>
		</init-param>
		...		
	</servlet>
...

That’s it. All configuration is done. Now, you just need to annotate you Resources. Let’s see how.

Say, you have a BookResource which exposes APIs to create/edit/list books. And you want
1. “Editor/Contributor” to be able to create a book, (@RolesAllowed({ “Editor”, “Contributor” }))
2. Only “Editor” to be able to edit a book, (@RolesAllowed({ “Editor” }))
3. While, anyone can see all the books. (@PermitAll)

Also, let’s say editing a book was released in API 1.3, so any client still using older APIs should not be able to update (@Version(“>1.3″)) an existing book. (Assuming you implemented VersionFilter properly)

Additionally, create or edit book should be audited with respective notes “create-book” and “edit-book” given your AuditFilter is in place.

package com.strumsoft.api;

import javax.ws.rs.FormParam;
import javax.ws.rs.GET;
import javax.ws.rs.POST;
import javax.ws.rs.PUT;
import javax.ws.rs.Produces;
import javax.ws.rs.core.Context;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.SecurityContext;

import com.flockthere.api.audit.Audit;

/**
 * Book Resource
 * 
 * @Path("/book/")
 * @author "Animesh Kumar <animesh@strumsoft.com>"
 * 
 */
@Produces({ MediaType.APPLICATION_XML, MediaType.APPLICATION_JSON })
public interface BookResource {

	// Only Editor and Contributor can create a book entry
	@javax.annotation.security.RolesAllowed({ "Editor", "Contributor" })
	@Audit("create-book")  // Audit
	@POST
	public Book createBook(@Context SecurityContext sc, @FormParam("title") String title,
			@FormParam("author") String author, @FormParam("publisher") String publisher,
			@FormParam("isbn") String isbn, @FormParam("edition") String edition);

	// Only Editor can edit an existing book entry
	@javax.annotation.security.RolesAllowed({ "Editor" })
	@Audit("edit-book")   // Audit
	@Version(">1.3")  // Available only on API version 1.3 onwards
	@PUT
	public Book editBook(@Context SecurityContext sc, @FormParam("title") String title,
			@FormParam("author") String author, @FormParam("publisher") String publisher,
			@FormParam("isbn") String isbn, @FormParam("edition") String edition);
	
	// Anyone can see these books
	@javax.annotation.security.PermitAll
	@GET
	public Book listBooks();

}

And at last, you will need to add jsr250-api to your project dependencies which defines javax.annotation.security.* annotations.

	<!-- securty tags: javax.annotation.security.* (@RolesAllowed, @PermitAll, @DenyAll etc.) -->
	<dependency>
		<groupId>javax.annotation</groupId>
		<artifactId>jsr250-api</artifactId>
		<version>1.0</version>
	</dependency>

Chill. :)

Written by Animesh

March 2, 2012 at 7:19 pm

Posted in Technology

Tagged with , , , , ,

AspectJ + Spring for method click records

leave a comment »

In one of the projects we recently did, we had a Spring and Jersey based JSON/XML REST API server. Clients would connect to this API server and access various resources. Now, for one of the reporting scenario, we needed to log who accessed which API method and when. This appeared to be very simple initially. I decided to intercept every http call, and log that.

However, when I dug deep, I found that one method might call another method internally, depending upon the situation. For example, say User-A is trying to login from Device-1, and he is already logged on to some other device, say Device-2. So, LoginResource would make a call to LogoutResouce to invalidate User-A’s session with Device-2. We couldn’t track this just by intercepting http calls. We needed to adopt some other strategy. Initially, I thought of logging each API method with brute force… but that was too inelegant to actually code. Then I looked at AspectJ, and I was enlightened.

First, Added AspectJ dependencies to pom.

...
	<dependencies>
		<!-- aspectj -->
		<dependency>
			<groupId>org.aspectj</groupId>
			<artifactId>aspectjtools</artifactId>
			<version>1.6.8</version>
		</dependency>
		<dependency>
			<groupId>org.aspectj</groupId>
			<artifactId>aspectjrt</artifactId>
			<version>1.6.8</version>
		</dependency>
		...
	<dependencies>
...

And updated spring config file.

<beans ... xsi:schemaLocation="... http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop.xsd ..."
 xmlns:aop="http://www.springframework.org/schema/aop" ... >
...
<aop:aspectj-autoproxy proxy-target-class="true" />
...

Then created a Java Method Annotation class.

import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

@Retention(RetentionPolicy.RUNTIME)
@Target({ ElementType.METHOD })
public @interface RecordClick {
	String action();  // Name of the action
	String comment() default ""; // Comment, if any?
}

Every method we want to track, I just need to annotate it with @RecordClick. Like this:


// API to signup
@RecordClick(action = "signup", comment = "User initiated signup")
public User explicitSignup(String firstName, String lastName, String email, String phone, String password, Role role) throws ValidationException {
}

// API to signup
@RecordClick(action = "signup", comment = "System initiated signup")
public User implicitSignup(String email, String phone) throws ValidationException {
}

// API to signin
@RecordClick(action = "signin", comment = "User initiated signin")
public Session explicitSignin(String username, String password) {
}

Okay, so now I have everything in place. I just need to create an Aspect to advice pertinent methods. So, I will create an Aspect class. This will intercept all public methods that are annotated with @RecordClick annotation, and create a click record for this call.

@Aspect
@Component
public class RecordClickMethodInterceptor {

	private static final Logger log = LoggerFactory.getLogger(RecordClickMethodInterceptor.class);

	// This will create and save an individual click
	@Autowired
	private ClickResource clickResource;

	/**
	 * Intercepts all public methods annotated with @RecordClick
	 *
	 * @param pjp
	 * @param recordClick
	 * @return
	 * @throws Throwable
	 */
	@Around("execution(public * *(..)) && @annotation(recordClick)")
	public Object doClickRecord(ProceedingJoinPoint pjp, RecordClick recordClick) throws Throwable {
		log.info("Intercepted method={} with action={}", pjp.getSignature().getName(), recordClick.action());

		List<String> paramNames = new ArrayList<String>();
		try {
			MethodSignature signature = (MethodSignature) pjp.getSignature();
			// Fetch method argument names.
			// Note: to enable this, you must compile the code in debug mode.
			paramNames = Arrays.asList(signature.getParameterNames());
		} catch (Exception e) {
			log.error("Can't record clicks. You must compile the code with debug=true");
			// Proceed with method call
			return pjp.proceed();
		}

		log.info("paramNames={}", paramNames);

		String user = "anonymous";
		Map<String, Object> methodArguments = new HashMap<String, Object>();

		int index = 0;
		// Iterate through method arguments
		for (Object arg : pjp.getArgs()) {
			// name of this argument
			String name = paramNames.get(index++);
			log.info("{} ==> {}", name, arg);

			// if arg is null, skip
			if (null == arg) {
				continue;
			}

			// Fetch user from argument '@Context SecurityContext'
			if (arg instanceof SecurityContext) {
				Principal principal = ((SecurityContext) arg).getUserPrincipal();
				if (null != principal) {
					log.info("Found user={}", principal.getName());
					user = principal.getName();
				}
			}
			// Fetch user from argument 'Principal principal;
			else if (arg instanceof Principal) {
				Principal principal = (Principal) arg;
				log.info("Found user={}", principal.getName());
				user = principal.getName();
			}
			// save to methodArguments
			else {
				methodArguments.put(name, arg.toString());
			}
		}

		// execute the method
		Object returned = pjp.proceed();

		// Process the returned value

		// Record this click
		String action = recordClick.action();
		String comment = recordClick.comment();
		clickResource.createClick(user, action, methodArguments, comment);
		return returned;
	}
}

I also need to compile the code in debug mode, unless I do this, I will not be able to fetch method argument names which I desperately need in order to make any sense of these records. So, I needed to update maven-compiler-plugin to enable debug mode.

<plugin>
	<artifactId>maven-compiler-plugin</artifactId>
	<configuration>
		<source>${java-version}</source>
		<target>${java-version}</target>
		<!-- compile with debug so as to get param names in aspects -->
		<debug>true</debug>
		<debuglevel>lines,vars,source</debuglevel>
	</configuration>
</plugin>

That’s it! have fun. :)

Written by Animesh

September 29, 2011 at 5:20 pm

In Studio: Javascript Primitives

with 2 comments

Javascript – and for that matter most other languages – has two kinds of stuffs: primitives and objects. Primitives are immutable and stored in a variable entirely. That is, when you do assignments among primitives, the assigned receives a copy from the assigner.

y = x; // y receives a copy of x.

Objects are not stored in the variable. They stay somewhere else, and the variable only holds a reference to that location. So, in case of object assignments only references get copied.

obj2 = obj1; // obj2 receives a copy of reference to the object referred by obj1.

That is, both obj1 and obj2 now points to the same object, and if you change this object – the change will be observed by both obj1 and obj2.

In Javascript, there are 5 kinds of primitives: undefined, null, boolean, string and number. So, string “abc”, number 1, 1.34, boolean true, false – these all are primitives. Rest everything are Objects.

I know what you are thinking? Primitives are not Objects, and so they must not be able to execute any method on themselves – because they don’t have any – then how come this works?

console.log(“abc”.length); // prints 3

Well, the reason this works is that Javascript wraps these primitives with their Object counterparts.

“abc” ==> new String(“abc”)
true ==> new Boolean(true)
1 ==> new Number(1)

Recall Java’s wrapper objects? Well, this is almost the same and with more juicy auto-boxing in play. Javascript automatically coerces between primitives and Objects. In this case,

  1. string value is coerced into a String Object,
  2. property ‘length’ is accessed, and then
  3. the coerced Object is discarded for garbage collection!

Now, the hacker in you must have started thinking: If Javascript automatically coerces primitives to Objects; then certainly, you should be able to assign properties to a primitive too, like this:

var my_bank_account = “State Bank of India”;
my_bank_account.city = “Indore”;
console.log(my_bank_account.city); // prints undefined.

Oops. It failed. But why?

Well, what you guessed was right. The moment you tried to assign a property to the primitive, Javascript did indeed coerce primitive string into the wrapper Object – in this case, String. This new Object got itself assigned a brand new property – ‘city’ – too. But, since there wasn’t any placeholder to store the reference to this new Object, it was quickly discarded and sent for garbage collection. And when you went back to access the property ‘city’, Javascript again coerced your primitive to the wrapper Object – which incidentally is a new Object and doesn’t know anything about your ‘city’ property hence you see ‘undefined’.

Let me show you – roughly – what happens behind the scene:

var my_bank_account = “State Bank of India”;
my_bank_account.city = “Indore”;  ==> (new String(“State Bank of India”)).city = “Indore”;
console.log(my_bank_account.city);  ==> console.log(  (new String(“State Bank of India”)).city );

This whole game seems messy? No. How do I know when am I dealing with primitives and when with Objects?

Well, you use ‘typeof’ operator.

typeof false; // ‘boolean’
typeof new Boolean(false); // ‘object’
typeof “Animesh”;// ‘string’
typeof new String(“Animesh”); // ‘object’
typeof 1.45; // ‘number’
typeof new Number(1.45); // ‘object’

Enlightened? Good. :)

One more thing: this entire coercion business is a double-way traffic. The way Javascript turns your primitives into Objects when needed, it can turn Objects into primitives too when required.

var s = new String(‘hello’);
typeof s; // ‘object’
var p = s.valueOf(); // converts to primitive
typeof p; // ‘string’

Javascript will automatically use ‘valueOf’ (and sometimes, ‘toString’) method whenever such a need arises. Watch this:

var x = new String(“abc”);
typeof x; // ‘object’
var y = x + 1; // automatically calls valueOf() on Object x.
typeof y; // ‘string’

Normally, you don’t need to worry about this thing. But knowing what goes on certainly helps at times. Don’t you think? Follow this:

var b = new Boolean(false);
typeof b; // ‘object’
if (b){
    console.log(“b is true.”); // Ah! When did b become true?
}

What’s wrong with the above code? You must have thought that Javascript would coerce the Boolean object into primitive while evaluating ‘if’? Well, Javascript didn’t think so. It just went ahead and evaluated the Object directly for the condition… and since that Object was neither null nor undefined, it evaluated to true.

If you had done this:

if (b.valueOf(){
    console.log(“b is true”);
} else {
    console.log(“b is false”); // thank God!
}

It would have come out differently. Remember this: unless there is a dire need, Javascript doesn’t perform any coercion.

I hope this article was helpful. I’ll be writing more on Javascript internals.

Written by Animesh

June 22, 2011 at 11:49 pm

Posted in Technology

Tagged with , , ,

Fun with Singleton (Python, Javascript, Java)

leave a comment »

They say that Singletons, like global variables, are evil. They hide dependencies; are harder to test and even harder to extend. Singletons are lies, and it’s best to keep away from them. But, there are scenarios where you need them. For example, when you want a shared resource like printer spooler or file manager or log manager, you want a single object to handle requests from all the various parts of your application.

In this blog, I am going to explore various ways to make Singletons in languages like Python, Java and Javascript so as to keep it simple, elegant and usable. Let’s talk about Python first. I love it, and it’s a really really wonderful language, and in here, there are n different ways to solve a problem. Singletons are no exception. The most natural way to do it is to create a decorator.

class Singleton(object):
	def __init__(self, klass):
		self.klass = klass   # class which is being decorated
		self.instance = None  # instance of that class
	def __call__(self, *args, **kwargs):
		if self.instance is None:
			# new instance is created and stored for future use
			self.instance = self.klass(*args, **kwargs)
		return self.instance

Now, let’s say you have a Resource class. To make it singleton, you just need to decorate it with ‘@Singleton‘, and you are done.

@Singleton
class Resource(object):
	def __init__(self, klass):
		self.name = None

Cool…eh? There are other – nerdy – ways too. Python uses an internal dictionary ‘__dict__’ variable to keep track of an Object’s properties and methods. So, if you can share ‘__dict__‘ across multiple instances of a Class, you can share the state between them. And isn’t that Singleton? Yes, that is. You might have many many instances, but all of them behave exactly the same.

class Singleton(object):
	_shared = {}
	def __init__(self):
		self.__dict__ = Singleton._shared
class Resource(Singleton):
	def __init__(self, klass):
		self.name = None

Since ‘self.__dict__‘ now refers to ‘_shared‘ dictionary, all instances of Resource would use the same dictionary and hence they will all have the same behavior. Geeky? Let me show you an even geekier way to do it.

In Python, when you instantiate a class, the interpreter calls ‘__new__‘ method – a class method which returns an instance of the class – and then ‘__init__‘ method – constructor of the class – is called which initializes the object. So, you can hack into ‘__new__‘ and return the single instance whenever it is being called.

class Singleton(object):
	_instance = None
	def __new__(cls, *args, **kwargs):
	# cls is the Class and the rest are constructor arguments
		if cls._instance is None:
			# create an instance and store it
			cls._instance = Object.__new__(cls, *args, **kwargs)
		return cls._instance
class Resource(Singleton):
	def __init__(self, klass):
		self.name = None

Aweomse! Isn’t it? There are other ways that deal with ‘__metaclass__‘ etc. but let’s save them for another day. Let’s use it now:

# get resource r1
r1 = Resource();
# get resource r2  (since Resource is singleton, r1 == r2)
r2 = Resource();
# to verify, let's set 'name' onto r1
r1.name = "Animesh Kumar"
print r1.name
# and the same 'name' appears in r2 as well!
print r2.name

Let’s now see how do we do this in Javascript. For the most simple form, just define an Object Literal, and you are done.

var resource = {
	getName : function() {
		return this.name;
	},
	setName: function(name){
		this.name = name;
	}
}

Easy. You have an object which you can now share across your application modules and it just works. For more complex scenarios, like private variables and all, you might have to resort to something like this:

// self-executable wrapper function
var Resource = (function(){
	// Resouce class which is to made 'singleton'
	function _Resource() {
    	var name; // private variable
    	this.getName = function() {	// getter
    		return this.name;
    	};
		this.setName= function(name){ // setter
			this.name = name;
		};
		// do more stuffs
    }
    // instance holder
    var instance = new _Resource();</p>
    // return an object with 'getInstance' method
    return = {
        getInstance: function(){
            return instance;
        }
   };
})();

_Resource (line-04) is your function of interest, and you want to make it singleton. So, you create another function ‘Resource‘ which wraps over _Resource and returns an object with method ‘getInstance‘ which would return the same instance of _Resource every time it will be called.

Let’s try to use it now:

// get resource r1
r1 = Resource.getInstance();
// get resource r2  (since Resource is singleton, r1 == r2)
r2 = Resource.getInstance();
// to verify, let's set 'name' onto r1
r1.setName("Animesh Kumar");
console.log(r1.getName());
// and the same 'name' appears in r2 as well!
console.log(r2.getName());

So it was easy. No? Great.

Now, Java. I know every one of you would already know it. I would write about it anyway, just for the sake of completeness. In Java, you create a private static instance of the class, and use that instance wherever necessary.

public class Resource {
	// static instance (Note: Resource instantiation is done here, not in getInstance)
	private static Resource instance = new Resource();
	// property
	private String name;
	// private constructor to disable 'new'
	private Resource() {
	}
	// public staic method to get an instance of this class
	public static Resource getInstance() {
		return instance;
	}
	// getter
	public String getName() {
		return name;
	}
	// setter
	public void setName(String name) {
		this.name = name;
	}
}

Now, let’s use it.

	public static void main(String[] args) {
		// get resource r1
		Resource r1 = Resource.getInstance();
		// get resource r2  (since Resource is singleton, r1 == r2)
		Resource r2 = Resource.getInstance();
		// to verify, let's set 'name' onto r1
		r1.setName("Animesh Kumar");
		System.out.println(r1.getName());
		// and the same 'name' appears in r2 as well!
		System.out.println(r2.getName());
	}

Loud and clear. And this also stops you from instantiating ‘Resource‘ with ‘new‘ operator. Try this:

	Resource r1 = new Resource();  // java.lang.Error: Unresolved compilation problem:

Your code will not compile. I know you know why. I will write anyways: because the constructor is private! So, there is no way to get an instance of Resource class but through ‘getInstance’ method which ensures single instance of the class. Also, If you noticed: I have instantiated ‘instance’ during declaration itself, not in ‘getInstance’ method. This way, the object gets created at the time of class loading, and you save yourself from a lot of issues that creeps in because of Java Just in Time.

Written by Animesh

May 27, 2011 at 3:59 pm

Follow

Get every new post delivered to your Inbox.

Join 205 other followers