CSV to Chart Utility

In my last project we had a requirement where we need to build the good report from the csv data. One way to do that is to use excel and create chart from excel. 
But to combine all chart in single report(If there are multiple charts), do data processing if any (We can use micro if person is good in excel) is tedious job.  I am attaching the mevenize project with some screenshot which will help you to understand the Utility.

This tool is helpful to create good charts/reports from CSV. You can pass the set of CSV, tool will do the data processing(data conversion)
if defined and create a pdf/xls/doc based charts on user choice. Tool will create a chart and tabular report as configuration and 
render that in a page. For all valid CSV’s, it will create a chart report in a new page.

Underneath it is using Dynamic Reporting for charting/reporting and Spring DI for report configuration. All report configuration specified below can be configured using XML file. 

1.       Name of PDF (refer attachment Report Config.png)

2.       Report header (refer attachment ReportConfig.png)

3.       Report footer

4.       Chart name (refer attachment Component-Config.png)

5.       Chart X and Y axis (refer attachment Component-Config.png)

6.       Chart Label (refer attachment Component-Config.png)

7.       Chart Type (refer attachment Component-Config.png) - Currently It support Bar, Line,XYLine chart.

8.       Blank page (refer attachment BlankPageConfig.png)

9.       Data Processor – Data processor is used to convert Raw data into some specified format which should be shown on report like change milliseconds to second, hours to min, Byte to MegaByte, etc (refer attachment DataProcessor.png)

Some configuration for Report Component
Report Configuration
       <bean id="performanceReport" class="com.sapient.dynamicreports.configuration.Report">
              <property name="name" value="PerformanceReports" /> - output report file name
              <property name="displayName" value=" Performance Test Report  " /> - output report header
              <property name="outputType" value="PDF" /> - output report type
              <property name="outputPath" value="c:\\output" /> - output path
              <property name="components">
                     <list>
              List of components ---
                           <!-- <ref bean="blankPage" /> -->
                           <ref bean="cpuUtilization1" />
                           <ref bean="memoryUtilization1" />
                           <ref bean="threadUtilization1" />

                           <ref bean="cpuUtilization2" />
                           <ref bean="memoryUtilization2" />
                           <ref bean="threadUtilization2" />

                           <ref bean="cpuUtilization3" />
                           <ref bean="memoryUtilization3" />
                           <ref bean="threadUtilization3" />

                           <ref bean="responseTimePerTransaction" />

                     </list>
              </property>

 Component Configuration

       <bean id="activeThreadOverTime" class="com.sapient.dynamicreports.configuration.Component">
              <property name="name" value="ActiveThreadOverTime" />

-    Name of the component

              <property name="displayName" value="Active Users Over Time" />

-    Display name , which will come in the output report

              <property name="dataSource" value="CSV" />

-    Data source type – in our case it will be CSV

              <property name="dataSourcePath"
                     value="c:\\csvfileset\\input\\ActiveThreadsOverTime.csv" />

-    Data source – CSV file name

              <property name="dataSourceFileName" value="ActiveThreadsOverTime.csv" />
              <property name="chartConfiguration" ref="atotChartConfiguration" />

-    Define the chart Configuration

              <property name="pageType" value="A4" />

-    Page type of chart

              <property name="pageOrientation" value="PORTRAIT" />

-    Page orientation of chart

              <property name="dataProcessor" ref="customDataProcessor" />

-    Data processor if any

              <property name="tableType" value="NO" />

-    Table type could be NO, AGGREGATED, SIMPLE

       </bean>


       </bean>

Some of the screenshot (ReportPage1.png, ReportPage2.png) will give an idea how the report will look like. also fine high level class diagram for code in Image1 and Image2.  You can download code from here

JQuant - Library for Financial/Mathematical Projects.

While Surfing over the internet comes accross the utility called JQuant,
 Another thought strikes in my mind is 'lets share it with everone using Blog'.
Its a very useful utility for Finanacial Project and Supports with lots of mathematical function. Basically JQuant is derived from the QuantLib  which is written in C++.


JQuantLib provides a free, open-source and comprehensive framework for quantitative finance. 
It's a 100% Java translation of QuantLib, which is written in C++. JQuantLib provides pricing valuation of a wide range of asset classes,
methods and models.


The Basic feature which JQuant supports is 


Features

•  Support to a wide range of financial instruments, including but not limited to European Options, American Options, Bermudan Options, Asian Options, Bonds, Swaps, FRA, Repo, Cap/Floors, etc
•  Several pricing engines: Black-Scholes, Barone-Adesi-Whaley, Bjerksund-Stensland, Ju Quadratic, Integral, Binomial Cox-Ross-Rubinstein, Binomial Jarrow-Rudd, Additive EquiProbabilities, Binomial Trigeorgis, Binomial Tian, Binomial Leisen-Reimer, Binomial Joshi, Finite Differences
• Implemented as Java Library with minimum external dependencies
• Coded with performance in mind. GC imposes minimalist performance penalty.

Capability of QuantLib
The Wiki Page gives the complete descrition of QuantLib and what all can bedone with the supported library.

I will keep you posted on other tool/utility which can help us in our day to day projects.

Application Monitoring

For few days i was looking to develop something which can monitor tomcat and tomcat activities.Monitor the tomcat application at client side is something can help us. Using Jconsole/jvisualvm using JMX/Mbean is one way to monitor the application but that has limited feature with limited UI. Something which is rich in UI and easy to use can really helpful.

I have checked and found few thing 

Different ways for Application Monitoring

This link tell us different way to monitor the web application. What i like here is  Psi-probe.

Probe Feature

Its simple to use to start with. Just put the probe.war in tomcat directory and configured the tomcat user in tomcat-users.xml and you are ready to go. 

Unlike many other server monitoring tools, PSI Probe does not require any changes to your existing apps. It provides all of its features through a web-accessible interface that becomes available simply by deploying it to your server. These features include:

• Requests: Monitor traffic in real-time, even on a per-application basis.
• Sessions: Browse/search attributes, view last IP, expire, estimate size.
• JSP: Browse, view source, compile.
• Data Sources: View pool usage, execute queries.
• Logs: View contents, download, change levels at runtime.
• Threads: View execution stack, kill.
• Connectors: Status, usage charts.
• Cluster: Status, usage charts.
• JVM: Memory usage charts, advise GC
• Java Service Wrapper: Restart JVM.
• System: CPU usage, memory usage, swap file usage. 

 I will keep posted to other application which can help in monitor.

Example Java

Here is the good links to understand and learn java in good way

http://www.exampledepot.com/egs/index.html

http://www.java2s.com

CSV Data Loader

Here I am sharing quick CSV Data Loader using scanner.

import java.io.File;
import java.io.FileNotFoundException;
import java.util.Scanner;

public class DataLoader {

public void laodData(String fileName) {
Scanner scanner = null;
try {
scanner = new Scanner(new File(fileName));
} catch (FileNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}

while (scanner.hasNextLine()) {
String line = scanner.nextLine();
System.out.println("CSV Line : " + line);
Scanner scanner2 = new Scanner(line);
scanner2.useDelimiter(", *");

// Read and sum numbers.
while (scanner2.hasNext()) {
System.out.println(scanner2.next());
}
System.out.println("________________ LINE END ____________________");

}
}

public static void main(String[] args) {
new DataLoader().laodData("c:\\test3.csv");
}
}

Apache Netty

While working on Server Client Communication Framework i came accross the Apache Netty which is pretty much the same I am plannign to develop.
Here is basic example of using these API provided by apache
http://bruno.factor45.org/blag/2010/07/15/handshaking-tutorial-with-netty/

We also need to see Apache MINA.

Here is a pretty good comparision between these 2

http://stackoverflow.com/questions/1637752/netty-vs-apache-mina


I am also exploring these 2 and will post some more details soon.

Thread Helper

Thread Helper will help you to call your class method in threaded environment. It will take care all the low level complexity and provide you a simple interface to call a method in threaded environment.
To use this one should extend the class called ThreadCommand and implement the execute method of that. Whatever code will be written in execute method will be called in threaded manner.

ThreadHelper has 2 methods in it
1. Call
2. Get

Call will notify the thread helper to call the execute method of Command implementation and store the result in a map and when get method will be called, result will returned to the user and it will be removed from the map.


After call method, user can do other stuff and whenever user required the output from thread just get that from get Method. This will simulate the post evaluation mechanism. In this user first call and continue with other operation and whenever required the output just call the get method.

ThreadHelper.java

import java.util.HashMap;

import java.util.Map;

import java.util.concurrent.ExecutionException;

import java.util.concurrent.ExecutorService;

import java.util.concurrent.Executors;

import java.util.concurrent.Future;



public class ThreadHelper

{

 private boolean callWhenFetch = false;

 private ExecutorService executorService = null;

 private Map> futurs = null;

 private boolean started = false;



 /**

  *

  * @param callWhenFetch

  */

 public ThreadHelper(boolean callWhenFetch)

 {

  this.callWhenFetch = callWhenFetch;

 }



 /**

  *

  */

 public ThreadHelper()

 {

 }



 public void start()

 {

  executorService = Executors.newCachedThreadPool();

  futurs = new HashMap>();

  started = true;

 }



 /**

  * Check if the thread Helper is Started

  *

  * @return

  */

 public boolean isStart()

 {

  return started;

 }



 /**

  * Initiates an orderly shutdown in which previously submitted commands are

  * executed, but no new commands will be accepted.

  */

 public void stop()

 {

  if (futurs != null)

  {

   futurs = null;

  }



  if (executorService != null)

  {

   executorService.shutdown();

  }

 }



 /**

  * Attempts to stop all actively executing commands, halts the processing of

  * waiting commands

  */

 public void stopNow()

 {

  if (futurs != null)

  {

   futurs = null;

  }



  if (executorService != null)

  {

   executorService.shutdownNow();

  }

 }



 /**

  * This method takes ICommand as input and submit to ThreadPool.

  *

  * @param command

  * @throws Exception

  */

 public synchronized void call(ThreadCommand command) throws Exception

 {

  checkIfStarted();

  Future submitted = executorService.submit(command);

  futurs.put(command.getId(), submitted);

 }



 /**

  * This method give the output to previously called command. If

  * callWhenFetch parameter is set to true then If you haven't call call

  * method before get then it will first call call method and then call get

  * of that call. If flag is set to false and you tried to call get before

  * call method it will throws exception

  *

  * @param command

  * @return

  * @throws NotNotifiedException

  * @throws InterruptedException

  * @throws ExecutionException

  */

 public T get(ThreadCommand command)

   throws NotNotifiedException,

   InterruptedException,

   ExecutionException,

   Exception

 {

  checkIfStarted();

  Future submitted = null;

  try

  {

   synchronized (futurs)

   {



    if (!futurs.containsKey(command.getId()))

    {

     if (!callWhenFetch)

     {

      throw new NotNotifiedException(

        "Object is not called before get.");

     }

     submitted = executorService.submit(command);

     futurs.put(command.getId(), submitted);

    }

    T object = futurs.get(command.getId()).get();

    submitted = futurs.remove(command.getId());



    return object;

   }

  }

  finally

  {

   if (submitted != null)

   {

    submitted.cancel(false);

    submitted = null;

   }

  }

 }



 private void checkIfStarted() throws Exception

 {

  if (!isStart())

  {

   throw new Exception(

     "Please start the thread helper before using it.");

  }

 }



 public int waitingCallCount()

 {

  return futurs.size();



 }

}

ThreadCommand.java

import java.util.concurrent.Callable;



public abstract class ThreadCommand implements Callable

{



 private String id;



 public ThreadCommand(String id)

 {

  this.id = id;

 }



 public T call() throws Exception

 {

  return execute();

 }



 public String getId() {

  return id;

 }



 public abstract T execute();



 @Override

 public String toString()

 {

  return "ThreadCommand [id=" + id + "]";

 }





}

ThreadHelperTest.java

import java.util.ArrayList;

import java.util.List;



import junit.framework.TestCase;






public class ThreadHelperTest extends TestCase

{



 public void testSingleThreadImmediateResponseBefoereStart()

 {

  try

  {

   ThreadCommand command = new ThreadCommand("Test")

   {



    @Override

    public String execute()

    {

     // TODO Auto-generated method stub

     return "I am just checking how execute is working.";

    }

   };



   ThreadHelper helper = new ThreadHelper();

   helper.call(command);

   helper.get(command);

   fail();

  }

  catch (Exception e)

  {

   assertEquals("Please start the thread helper before using it.", e

     .getMessage());



  }

 }



 public void testGetBeforeCallWhenCallWhenFetchIsFalse()

 {

  try

  {

   ThreadCommand command = new ThreadCommand("Test")

   {



    @Override

    public String execute()

    {

     // TODO Auto-generated method stub

     return "I am just checking how execute is working.";

    }

   };



   ThreadHelper helper = new ThreadHelper(true);

   //helper.call(command);

   helper.get(command);

   fail();

  }

  catch (Exception e)

  {

   e.printStackTrace();

   assertEquals("Please start the thread helper before using it.", e

     .getMessage());



  }

 }



 public void testSingleThreadImmediateResponse()

 {

  try

  {

   ThreadCommand command = new ThreadCommand("Test")

   {

    public String execute()

    {

     return "I am just checking how execute is working.";

    }

   };



   ThreadHelper helper = new ThreadHelper();

   helper.start();

   helper.call(command);

   String str = helper.get(command);

   assertEquals("I am just checking how execute is working.", str);



  }

  catch (Exception e)

  {

   fail();



  }

 }



 public void testMutipleThreadImmediateResponse()

 {

  try

  {

   ThreadCommand command1 = new ThreadCommand("Test1")

   {

    public String execute()

    {

     return "[Test 1]I am just checking how execute is working.";

    }

   };



   ThreadCommand command2 = new ThreadCommand("Test2")

   {

    public String execute()

    {

     return "[Test 2]I am just checking how execute is working.";

    }

   };



   ThreadHelper helper = new ThreadHelper();

   helper.start();

   helper.call(command1);

   helper.call(command2);

   String str = helper.get(command2);

   assertEquals(

     "[Test 2]I am just checking how execute is working.",

     str);

   str = helper.get(command1);

   assertEquals(

     "[Test 1]I am just checking how execute is working.",

     str);



  }

  catch (Exception e)

  {

   fail();



  }

 }



 public void testMutipleThreadImmediateResponseForSameCommand()

 {

  try

  {

   ThreadCommand command1 = new ThreadCommand("Test1")

   {

    public String execute()

    {

     return "[Test 1]I am just checking how execute is working.";

    }

   };



   ThreadHelper helper = new ThreadHelper();

   helper.start();

   helper.call(command1);

   String str = helper.get(command1);

   assertEquals(

     "[Test 1]I am just checking how execute is working.",

     str);

   str = helper.get(command1);

   fail();



  }

  catch (Exception e)

  {

   assertEquals("Object is not called before get.", e.getMessage());



  }

 }



 public void testThreadHelperStatusAfterAllGet()

 {

  try

  {

   List> commands = new ArrayList>();

   for (int i = 0; i < k =" i;"> command1 = new ThreadCommand(

      "Test" + k)

    {

     public String execute()

     {

      return "[Test "

        + k

        + "]I am just checking how execute is working.";

     }

    };

    commands.add(command1);

   }



   ThreadHelper helper = new ThreadHelper();

   helper.start();

   for (ThreadCommand command : commands)

   {

    helper.call(command);

   }



   assertEquals(100, helper.waitingCallCount());



   // get all odd number command

   for (int i = 1; i < i =" i" i =" 0;"> command : commands)

   {

    if(i == 100) {

     break;

    }

    assertEquals("[Test "

      + i

      + "]I am just checking how execute is working.", helper

      .get(commands.get(i)));

    i = i + 2;

   

   }

   // get rest of the commands

   assertEquals(0, helper.waitingCallCount());



  }

  catch (Exception e)

  {

   e.printStackTrace();

   fail();



  }

 }

}

Eclipse Memory Analyzer

http://wiki.eclipse.org/index.php/MemoryAnalyzer

I will post the details soon

XLogger

Last few days I was looking to improve the logger. Main issue with the logger that we need to make sure we are using log.isXXXEnabled() for all log calls. So we are making sure that in the log message we are not making unnecessary string if logger is not going to print that. Also sometime we do perfLogger with System.currentTimeInMillis but we actually not using in prod server, but this system call are very time consuming. I was thinking of how to improve it so i have come up with customize logger called XLogger.

XLogger.java

import java.text.MessageFormat;

import org.apache.log4j.LogManager;

import org.apache.log4j.Logger;

import org.apache.log4j.Priority;

/**

*

* XLogger is wrapper around the Logger provided by apache.

* This framework will make sure that you are not building a string for

* log until it is not candidate for logging. Logger has provided isXXXEnable method

* to do this but for every logging statement, using this is tedious.

*

* It is also helpful for performance logging.

* @author Viren.Balaut

*

*/

public class XLogger

{

private Logger logger = null;

private boolean isPerfLogger = false;

private long startTime = -1;

private long endTime = -1;

public static XLogger getLogger()

{

return new XLogger(false);

}

public static XLogger getLogger(String name)

{

return new XLogger(name, false);

}

public static XLogger getLogger(Class clazz)

{

return new XLogger(clazz, false);

}

public static XLogger getPerfLogger()

{

return new XLogger(true);

}

public static XLogger getPerfLogger(String name)

{

return new XLogger(name, true);

}

public static XLogger getPerfLogger(Class clazz)

{

return new XLogger(clazz, true);

}

private XLogger(boolean isPerfLogger)

{

logger = LogManager.getRootLogger();

this.isPerfLogger = isPerfLogger;

}

private XLogger(String name, boolean isPerfLogger)

{

logger = LogManager.getLogger(name);

this.isPerfLogger = isPerfLogger;

}

private XLogger(Class clazz, boolean isPerfLogger)

{

logger = LogManager.getLogger(clazz);

this.isPerfLogger = isPerfLogger;

}

public void start()

{

if (logger.isDebugEnabled() && isPerfLogger)

{

startTime = System.currentTimeMillis();

}

}

public void stop()

{

if (logger.isDebugEnabled() && isPerfLogger)

{

if (startTime <= -1)

{

throw new IllegalStateException(

"Please call start() before calling end.");

}

endTime = System.currentTimeMillis();

}

}

public void perf(String message, String methodName)

{

if (logger.isDebugEnabled() && isPerfLogger)

{

if (startTime <= -1 endTime <= -1)

{

throw new IllegalStateException(

"Please call start() and end() both before calling logPerf.");

}

debug(

message + ", Time taken({0}) in MS : {1}",

methodName,

(endTime - startTime));

resetTimer();

}

}

public void console(String message, Object... objects)

{

System.err.println(getMessage(message, objects));

}

public void debug(String message, Object... objects)

{

if (logger.isDebugEnabled())

{

logger.debug(getMessage(message, objects));

}

}

public void info(String message, Object... objects)

{

if (logger.isInfoEnabled())

{

logger.info(getMessage(message, objects));

}

}

public void warn(String message, Object... objects)

{

if (logger.isEnabledFor(Priority.WARN))

{

logger.warn(getMessage(message, objects));

}

}

public void error(String message, Object... objects)

{

if (logger.isEnabledFor(Priority.ERROR))

{

logger.error(getMessage(message, objects));

}

}

public void error(String message, Throwable e, Object... objects)

{

if (logger.isEnabledFor(Priority.ERROR))

{

logger.error(getMessage(message, objects), e);

}

}

public void fatel(String message, Throwable e, Object... objects)

{

if (logger.isEnabledFor(Priority.FATAL))

{

logger.fatal(getMessage(message, objects), e);

}

}

private String getMessage(String message, Object... objects)

{

try

{

MessageFormat form = new MessageFormat(message);

String msg = form.format(objects);

return msg;

}

catch (Exception e)

{

System.err.println("Error occurred while parsing message for "

+ message);

return message;

}

}

private void resetTimer()

{

startTime = -1;

endTime = -1;

}

}

Resource Handling

Resource Handling
In Java, Most of the time we open some resource and after using we forget to close the resource and that make a lots of open connection for the resouse if we are not using a good ConnectionPool for resource. Still if we are using the connectino pool, then also we need to release the resopuse after use, so other can use that resource for their operation.
here is sample framework which basically takes care of these part in your java program.

Here I have two classes, one is TransactionGuard and other is Command interface.

TransactionGuard.java

/**
* TransactionGuard is make sure you release the particular resource after use.
* By using any resource, it will make sure that you are closing the resource
* after using. To use this one need to implement getConnection and
* releaseConnection method based on the need.
* 
* @author Virendra.Balaut
* 
* @param 
* an element/resource for which you want to implement
* TransactionGuard
*/
public abstract class TransactionGuard
{
/**
* This is the main execute method which takes care of getting a connection of T type,
* do some operation and after use close the connection.
* 
* @param command
* @throws Exception
*/
public void execute(Command command) throws Exception
{
T connection = null;
try
{
connection = getConnection();
command.execute(connection);
}
finally
{
releaseConnection(connection);
}

}

/**
* Abstract method to get Connection for a dedicated Type.
* 
* @return
*/
protected abstract T getConnection() throws Exception;

/**
* Abstract method to release connection for a dedicated type.
* 
* @param connection
*/
protected abstract void releaseConnection(T connection) throws Exception;
}

Command.java

public abstract class Command 
{
public abstract void execute();
}

Using these framework will ensure that resouce will automatically closed after use. 

Sample Example :-

Here I have sample program which will do database operation. So I need to open a database connection and after use it I need to release that connection.

new TransactionGuard()
{

@Override
protected Connection getConnection() throws Exception
{
Connection connection = createConnection();
return connection;
}

@Override
protected void releaseConnection(Connection connection)
throws Exception
{
if(connection != null) {
connection.close();
}

}
}.execute(new Command()
{

public void execute(Connection connection) throws Exception
{
// Do some operation with connection.

}
});

Let me know your suggestions/views.