Using Apache Thrift with Maven

When you use Thrift for your project, you have to manually generate the sources, put them inside your source folder and build.

This plugin does the work for you when you execute the maven build.

However, this plugin is not yet available on maven central repositories, therefore you have to add developer's repo. to your project pom.

 
    <pluginRepositories>
        <pluginRepository>
            <id>dtrotts</id>
            <name>Repository for maven thrift plugin</name>
            <url>http://maven.davidtrott.com/repository</url>
            <layout>default</layout>
        </pluginRepository>
    </pluginRepositories>
 

Then, as described in the plugin's project page, you have to:

1. Use Java 1.5 or newer due to the usage of Generics

You can specify it in the configuration of maven compiler plugin:

 
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>1.6</source>
                    <target>1.6</target>
                </configuration>
            </plugin>
            ...
 

2. Have Thrift executable is in your PATH or set the parameter of the plugin to the correct location.

 
            <plugin>
                <groupId>org.apache.thrift.tools</groupId>
                <artifactId>maven-thrift-plugin</artifactId>
                <version>0.1.10</version>
                <configuration>
                    <thriftExecutable>/usr/local/bin/thrift
                </configuration>
                ...
 

3. Include the dependency for libthrift

 
        <dependency>
            <groupId>org.apache.thrift</groupId>
            <artifactId>libthrift</artifactId>
            <version>0.8.0</version>
        </dependency>
 

Then place your *.thrift files to the directory: src/main/thrift.

Now you can execute mvn clean install as usual.

Volunteer Computing: An Introduction

As the technology advances and research areas widen, demand for computational power increases day by day. Such computational demands can be mainly observed in several categories. Physical simulations from molecular level to universe level, analysis of large data from optical telescopes, gene sequencers, gravitational wave detectors, particle colliders and biology-inspired algorithms are some of those categories.
These tasks require high-performance computing (HPC). So one solution is to use supercomputers. But typically, the rate of job completion is more important than the turnaround time of individual jobs since overall result is what’s useful. The term to refer to that idea is high-throughput computing.

To achieve high-throughput computing, distributed computing is a better approach since individual job can be processed in parallel in large quantities.
Available distributed computing options are:
○ Cluster computing - dedicated computers in a simple location.
○ Desktop grid computing - PCs within an organization as a computing resource.
○ Grid computing - sharing computing resources by separate organizations.
○ Cloud computing - a company selling access to computing power
○ Volunteer computing

Volunteer computing (also sometimes referred to as global computing) uses computational power volunteered by the general public to perform distributed scientific computing. Volunteers may include individuals as well as organizations such as universities.
This approach allow ordinary Internet users to volunteer their computer resources on idle time by forming parallel computing networks easily, quickly and inexpensively without needing expert help.

Typically when it comes to volunteer computing, the volunteers who contribute with their resources are considered to be anonymous, although some volunteer computing frameworks may collect information like a nickname and email address of the volunteers for the usage of credit system, etc.

Each of the distributed computing paradigms have different resources pools. For example, number of computers owned by a particular university when it comes to grid computing, and the number of servers owned by a company in cases of cloud computing. The number of total possible personal computers is the resource pool in the case of volunteer computing.

To understand the importance of volunteer computing, we have to consider its resource pool.
The number of privately-owned PCs around the globe is currently estimated as 1 billion and is expected to grow to 2 billion by 2015. Also, the resource pool is self-financing, self-updating and self-maintaining. Users buy and maintain their own computers. Therefore various costs associated with other types of grid computing do not apply to volunteer computing. Another important point is that consumer market adopts the latest technology quickly. A supercomputer or a computing grid cannot be replaced or upgradedeasily as newer technologies emerge. But the typical PC user can. For example, the fastest processors today are GPUs developed with computer games in mind. Due to these factors, we can state that volunteer computing has a huge potential for world computational needs.

Berkeley Open Infrastructure for Network Computing (BOINC) is the predominant volunteer computing framework in use.

Some of the other volunteer computing frameworks are:
○ Bayanihan Computing Group
○ JADIF - Java Distributed (volunteer / grid) computing Framework
○ Javelin Global Computing Project
○ XremWeb Platform
○ Entropia

Here is a list of most active volunteer computing projects as of January 2012.
○ SETI@home: Search for extra-terrestrial life by analyzing radio frequencies emanating from space
○ Einstein@home: Search for pulsars using radio signals and gravitational wave data
○ World Community Grid: Humanitarian research on disease, natural disasters, and hunger
○ Climateprediction.net: Analyse ways to improve climate prediction model
○ Folding@home: Computational molecular biology
○ LHC@home: Improve the design of the Large Hadron Collider and its detectors
○ Milkyway@home: Create a highly accurate three-dimensional model of the Milky Way galaxy using data collected from the Sloan Digital Sky Survey
○ Spinhenge@home: Study nano-magnetic molecules for research into localized tumor chemotherapy and micro-memory
○ PrimeGrid: Generate a list of sequential prime numbers, search for particular types of primes
○ Malariacontrol.net: Simulate the transmission dynamics and health effects of malaria

(This post includes citations from several sources and aims to summarize volunteer computing)

First Steps of Apache Thrift with Java in Linux

Apache Thrift is a software framework for scalable cross-language services development. It was originally developed by Facebook before it was donated to Apache Software Foundation.  

Download the stable release

Unpack the tar.gz archive to a directory you prefer
(say home /home/amila/apacheThrift)

tar -xvzf thrift-0.8.0.tar.gz

You need JDK and Apache Ant at least to run Thrift's Java tutorial.
(You can refer to my previous posts to find how to install JDK on ubuntu.)

Use apt-get to install Ant

sudo apt-get install ant
 

We first need to install Thrift compiler and language variables before we start developing we Thrift.

There are several required packages to install Thrift that are not installed on a linux distribution by default.
To install those

sudo apt-get install libboost-dev libboost-test-dev libboost-program-options-dev libevent-dev automake libtool flex bison pkg-config g++ libssl-dev 
 

Goto top level directory of unpacked thrift distribution
(eg: /home/amila/apacheThrift/thrift-0.8.0)

./configure
 

During this process, thrift will scan and list the different language found.
It should say:

Building Java Library ........ : yes
Using javac .................. : javac
Using java ................... : java
Using ant .................... : /usr/bin/ant

..along with other languages found on your computer.

However, to configure Thrift for all those languages, you may need to install additional packages

Now you can make Thrift:

make
 

You might get some error if all required libraries for the languages configured in above step are not present.

In that case, you can deselect the packages you don't need when configuring
For example, say you don't need the support for Ruby. When configuring, you can use:

./configure --without-ruby
 

(I had to deselect erlang libraries to get it working on ubuntu 11.04)

After make is completed successfully, install Thrift by,

sudo make install
 

To check if the installation is successfully completed:

thift
 

You should get an output like:

Usage: thrift [options] file
Options:
  -version    Print the compiler version
  -o dir      Set the output directory for gen-* packages
               (default: current directory)
  -out dir    Set the ouput location for generated files
.........
.........
 

Tutorial are located at ./tutorials directory.

There you will find two files tutorials.thrift and shared.thrift

.thrift files describe the interfaces (IDL) in terms of the classes, methods they include.

thrift -r -gen java tutorial.thrift
 

This will create a directory named "gen-java" inside your current directory which will include generated Java classes according to specified thrift file.

Now goto the directory "java" inside the current directory (tutorial) and execute ant.
The ant script will compile both generated source files and the source file inside java directory and build a jar file.

Finally, run the tutorial by:

thrift/tutorial/java$ ./JavaServer
thrift/tutorial/java$ ./JavaClient
 

You may also find this page useful.

Running / Debugging Apache Axis2 inside IntelliJ IDEA

1. Introduction
2. Required tools
3. Setting up
4. Getting the source and Building
5. Open Axis2 source with IDEA
6. Run/debug configuration
7. Add required libraries
8. Run/debug Axis2 within IDEA

1.Introduction

Apache Axis2 is a famous web services engine and toolkit which is heavily used in industry.



2.Required Tools

Linux
Sun JDK 1.6
IntelliJ IDEA 10.5
Maven 2
Subversion 1.6

3.Setting up

Make sure you've correctly installed Sun JDK, Maven2 and SVN. (refer to my previous posts for details)

4.Getting the source and Building


Create the folder where you want to get sources.
for example, if you want it in "axis2Source"

mkdir axis2Source
cd axis2Source

then get the source from apache subversion repository by,

svn checkout https://svn.apache.org/repos/asf/axis/axis2/java/core/trunk/ apache-axis2

goto the modules/distribution directory and execute:

mvn clean install -Dmaven.test.skip=true
 

That will compile the sources and build binary and source distributions.
Goto taget directory. You will see the binary distribution pack created as axis2-1.7.0-SNAPSHOT-bin.zi.
Extract it in the same directory.

5.Open Axis2 source with IDEA

Goto IDEA.
Goto File->Open Project
Browse to the axis2Source directory

Select the parent pom and press OK. IDEA will automatically import all the modules and create the project structure.


6.Run/debug configuration

Goto Run->Edit Configurations
Add "Application" configuration using the "+" button.
Select org.apache.axis2.transport.SimpleAxis2Server as the Mail class

Set the desired log4j properties file as a VM parameter:
For example:

-Dlog4j.configuration=file:/home/amila/asf/sources/axis2source/apache-axis2/modules/distribution/target/axis2-1.7.0-SNAPSHOT/conf/log4j.properties

You have to pass axis2 repository location and axis2.xml path as program parameters.

Set the working directory as extracted bin distro root

/home/amila/asf/sources/axis2source/apache-axis2/modules/distribution/target/axis2-1.7.0-SNAPSHOT

Select the module you want to debug in "Use classpath and JDK of module" and press OK

 
 7.Add required libraries

Then you have to make sure all the libs required for axis2 to start are available in the classpath.
To configure that, select the desired module (axis2kernal) in project window and press F4 to open configure dialog box.
Goto dependencies tab and press Add, select Library from drop down menu.

Click "New Library..."
Select "Attatch jar directories..."

Select the lib directory of the binary distribution

Give a name such as "axis2Lib" and press OK.
You can also select other modules as dependencies if you want to debug them as well.

Finally, press OK to exit the configuration dialog.

8.Run/debug Axis2 within IDEA

Now you can Start the Axis2 server inside IDEA by the Run command (Shift+F10).
Try pointing your browser to http://localhost:8080/ and see if services get listed.

And you can debug axis2 as you debug any other code. Try setting a breakpoint in SimpleAxis2Server and start debugging (Shift+F9)

Advantage here is that, you don't have to maven build every time you change something, as opposed to remote debugging.

XML Pretty Printing Without External Dependencies

This code uses javax.xml.transform to perform a simple XSL transformation to pretty print a given XML as a string. Hope someone will find this useful.

import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;

import javax.xml.transform.*;
import javax.xml.transform.stream.*;

public class XMLPrettyPrinter {
 
 public static void main(String[] args) {
  new XMLPrettyPrinter().demo();
 }

 private void demo() {
  String input = "info";
  String output = new String();
  try {
   output = this.prettify(input);
  } catch (Exception e) {
  }
  System.out.println("Input XML:\n" + input);
  System.out.println("\nOutput XML:\n" + output);
 }

 private String prettyPrintStylesheet = 
    ""
  + "  "
  + "  "
  + "  "
  + "    "
  + "  "
  + "  "
  + "        "
  + "          "
  + "        " 
  + "  " 
  + "";
 
 public String prettify(String inputXML) throws Exception {

  Source stylesheetSource = new StreamSource(new ByteArrayInputStream(
    prettyPrintStylesheet.getBytes()));

  Source xmlSource = new StreamSource(new ByteArrayInputStream(
    inputXML.getBytes()));
  ByteArrayOutputStream out = new ByteArrayOutputStream();

  TransformerFactory tf = TransformerFactory.newInstance();
  Templates templates = tf.newTemplates(stylesheetSource);
  Transformer transformer = templates.newTransformer();
  transformer.transform(xmlSource, new StreamResult(out));
  return out.toString();
 }

}

Building WSO2 Carbon from Source

1. Introduction
2. Required(and recommended) tools
3. Setting up
4. Getting the source
5. Building
6. Tips/Issues/Fixes

1.Introduction


Carbon is the core platform of complete enterprise middleware stack by WSO2 which is completely open source.
Read more here: http://wso2.org/



2.Required Tools

Linux
Sun Java Development Kit 1.6
Apache Maven 2.2
Apache Ant 1.8
Subversion 1.6

3.Setting up

You have to be online.

a.Install java:

Goto terminal

sudo add-apt-repository "deb http://archive.canonical.com/ lucid partner"
sudo apt-get update
sudo apt-get install sun-java6-jdk

Allow the download and accept the terms when done.
Environment variables will be automatically created for you by the installer.
Verify if it's correctly installed by

java -version
javac -version

b.Install Maven

sudo apt-get install maven2

Ant also should be installed automatically by maven.
Verify if it's correctly installed by

mvn -version
ant -version

c.Install Subversion

sudo apt-get install subversion

Verify if it's correctly installed by

svn help

d.Setting Maven option for permanent generation space

PermSize defines the size for the Permanent Generation which holds meta-data describing user classes. The Permanent Generation (PermGen) Space is set to a small value by default, by the JDK (64 MB maximum for JDK 1.5). Large applications such as Carbon with a large code base need this set to a higher value.
You need to set it in the environment variable MAVEN_OPTS
You can also increase the java heap size

MAVEN_OPTS=" -Xms512m -Xmx1024m -XX:MaxPermSize=1024m"
export MAVEN_OPTS

Better approach is to set these in .bashrc file in your home so the changes will not be lost when you begin another session (or terminal)

sudo gedit /home/amila/.bashrc 

Add the above too lines to the end of the file and save.

4.Getting the source
create the folder where you want to get sources.
for example, if you want it in "carbonSource"

mkdir carbonSource
cd carbonSource

then get the source from wso2 subversion repository by,

svn checkout http://svn.wso2.org/repos/wso2/trunk/carbon/

Wait until all the sources are downloaded. this includes all the dependencies
of carbon including several apache projects as well as all the carbon based
products. Therefore it's going to take a while.

5. Building
if you go inside the carbon directory, you can see





You can build all things by just using:

mvn install

That will compile the sources, build all dependencies, perform tests and build jar files.
But probably it will not work straightaway, specially in the first try. Also, testing
take a long time. We don't need to test everything when building for the first time.
Therefore we can skips the tests by setting the variable maven.test.skip to true.
You should build in the following order:

• dependencies
• orbit
• service-stubs
• core
• components
• features
• products

For example, to build dependencies:

cd dependencies/
mvn install -Dmaven.test.skip=true

When building, maven will automatically download files for plugins etc. needed
they are downloaded to local maven repository located at your home's /.m2/repository/

After you have successfully built dependencies, orbit, core, components and features, you can move on to building carbon

cd products/carbon/
mvn install -npu -Dmaven.test.skip=true 
 

You can also build other products if you like.


6.Tips/Issues/Fixes

Make sure you have plenty of time and bandwidth before starting to build
After building carbon for the first time, my m2 repository had 2GiB of downloaded content.
Skip tests when you build for the first time
If your build fails halfway, when you build again, use no plugin update flag (npu).
When it's used, maven won't check for updates on already downloaded packages,
only download missing packages, so it will speed up the process.

mvn install -npu -Dmaven.test.skip=true

Possible Issues & Fixes
==================


Sometimes build may fail complaining about missing artifact. Make sure your connection is working properly. Try again in a bit. If the missing artifact's has the "SNAPSHOT" part, it's a local file. Try to find the it's package using find command and build it from inside the particular package.
Compilation errors might occur rarely since carbon is under active development. If so, try updating the sources

svn up

You may also get the error:

[ERROR] FATAL ERROR
[INFO] ------------------------------------------------------------------------
Exception in thread "main" java.lang.OutOfMemoryError: PermGen space

This happens due to insufficient permanent generation space as described previously.
Verify you have correctly set maven options to increase permgen space by,

echo $MAVEN_OPTS

If it isn't correctly set, see the section "Setting Up" to set the variable.

Another possible error:

[ERROR] BUILD ERROR
[INFO] ------------------------------------------------------------------------
[INFO] Failed to create assembly: Failed to retrieve OS environment variables. Reason: Cannot run program "env": java.io.IOException: error=24, Too many open files

Increase the maximum number of open files to fix this

sudo gedit /etc/security/limits.conf 

Add these two lines to end of the file

* soft nofile 65536
* hard nofile 65536

You can save the terminal output for later reference with tee command
For example:

mvn install -npu -Dmaven.test.skip=true | tee /home/amila/BuildTry1.txt

Good Luck!

SIP Communicator

I'm sure all of you have used some form of instant messaging. Whether it's IRC, GTalk, Yahoo, Windows Live, ICQ, even facebook or any other service they're all instant messaging. It is usual for one to have several of these accounts. If you chat with contacts from those accounts it is easy to chat with all of them from a single place rather than logging in to all of them and use various software to chat. That's what a multi-protocol chat client does.
SIP Communicator is such software. You can add SIP, Jabber, AIM/ICQ, MSN, Yahoo! Messenger, IRC, Facebook and many other accounts to it. It also supports audio and video calls. Currently for SIP. It comes with some other cool features too.
And better yet, SIP Communicator is open source! You can contribute the development if you are interested in programming. Since it is developed using java. that means it supports variety of operating systems including linux, windows, mac os


you can take a look on more features here or download it.