Tug's Blog

Posted by Tug Grall at 2:12 AM 0 comments 👁 Image

MQTT (MQ Telemetry Transport) is a lightweight publish/subscribe messaging protocol. MQTT is used a lot in the Internet of Things applications, since it has been designed to run on remote locations with system with small footprint.

The MQTT 3.1 is an OASIS standard, and you can find all the information at http://mqtt.org/

This article will guide you into the various steps to run your first MQTT application:

1. Install and Start a MQTT Broker
2. Write an application that publishes messages
3. Write an application that consumes messages

The source code of the sample application is available on GitHub.

Posted by Tug Grall at 4:08 AM 0 comments 👁 Image

In this article you will learn how to create a MapR Streams Consumer that saves all the messages into a MapR-DB JSON Table.

Posted by Tug Grall at 12:06 AM 0 comments 👁 Image

You can find a new tutorial that explains how to deploy an Apache Kafka application to MapR Streams, the tutorial is available here:

• Getting Started with MapR Streams

MapR Streams is a new distributed messaging system for streaming event data at scale, and it’s integrated into the MapR converged platform. MapR Streams uses the Apache Kafka API, so if you’re already familiar with Kafka, you’ll find it particularly easy to get started with MapR Streams.

Posted by Tug Grall at 2:18 AM 0 comments 👁 Image

Ted Dunning and I have worked on a tutorial that explains how to write your first Kafka application. In this tutorial you will learn how to:

• Install and start Kafka
• Create and Run a producer and a consumer

You can find the tutorial on the MapR blog:

• Getting Started with Sample Programs for Apache Kafka 0.9

Posted by Tug Grall at 2:02 AM 0 comments 👁 Image

Posted by Tug Grall at 10:04 AM 1 comments 👁 Image

Moving my Java from Couchbase to MongoDB pic.twitter.com/Wnn3pXfMGi
— Tugdual Grall (@tgrall) January 26, 2015

• Moving the data from Couchbase to MongoDB
• Updating the application code to use MongoDB

Posted by Tug Grall at 7:01 AM 2 comments 👁 Image

Introduction

Design

• copy the versions the document into new documents
• copy the versions of the document into a list of embedded documents
• store the list of attributes that have been changed into a embedded element (or new documents)
• store the "delta"
• …

1. The current version is is a simple Key/Document, no change to the key.
2. The version is a copy of the document, and the version number is added to the key.

function (doc, meta) {
 if(doc.type && doc.type == "beer") {
 emit(doc.name);
 }
}

function (doc, meta) {
 if(doc.type && doc.type == "beer" && (meta.id).indexOf("::v") == -1 ) {
 emit(doc.name);
 }
}

 Object obj = client.get(key);
 if (obj != null) {
 // get the next version, create or use the key: mykey_version
 long version = client.incr(key + "_version", 1, 1); 
 String keyForVersion = key + "::v" + version; // mykey::v1
 try {
 client.set(keyForVersion, obj).get();
 } catch (Exception e) {
 logger.severe("Cannot save version "+ version + " for key "+ key +" - Error:"+ e.getMessage() );
 }
 }
 client.set(key, value);

 Object obj = client.get(key);
 // need to delete all the version first
 Object vObject = this.get(key + "_version");
 if (vObject != null) {
 long biggerVersion = Long.parseLong((String) vObject);
 try {
 // delete all the versions
 for (int i = 1; i <= biggerVersion; i++) {
 String versionKey = key + "::v" + i;
 client.delete(versionKey).get();
 }
 // delete the counter
 client.delete(key + "_version").get();
 } catch (InterruptedException e) {
 e.printStackTrace();
 } catch (ExecutionException e) {
 e.printStackTrace();
 }
 }
 client.delete(key);

Use versioning

git clone https://github.com/tgrall/couchbase-how-to-versioning.git
cd how-to-versioning
mvn clean install

...
 
 

...

 List

 client = new CouchbaseClientWithVersioning(uris, "default", "");
 client.setAutomaticVersionning(true);

Conclusion

Posted by Tug Grall at 6:59 AM 0 comments 👁 Image

Introduction

The Tool: Couchbase SQL Importer

+-------------+----------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------------+----------+------+-----+---------+----------------+
| ID | int(11) | NO | PRI | NULL | auto_increment |
| Name | char(35) | NO | | | |
| CountryCode | char(3) | NO | | | |
| District | char(20) | NO | | | |
| Population | int(11) | NO | | 0 | |
+-------------+----------+------+-----+---------+----------------+

city:3805
{ 
 "Name": "San Francisco",
 "District": "California",
 "ID": 3805,
 "Population": 776733,
 "CountryCode": "USA"
}

{
 "rows": [
 {"key": "city", "value": 4079}, 
 {"key": "country", "value": 239}, 
 {"key": "countrylanguage", "value": 984}
 ]
}

{
 "total_rows": 4079,
 "rows": [
 {"id": "city:1", "key": 1, "value": null}, 
 {"id": "city:2", "key": 2, "value": null}, 
 {"id": "city:3", "key": 3, "value": null}, 
 {"id": "city:4", "key": 4, "value": null},
 {"id": "city:5", "key": 5, "value": null}
 ]
}

{
 "total_rows": 984,
 "rows": [
 {"id": "countrylanguage:1", "key": ["ABW", "Dutch"], "value": null}, 
 {"id": "countrylanguage:2", "key": ["ABW", "English"], "value": null}, 
 {"id": "countrylanguage:3", "key": ["ABW", "Papiamento"], "value": null},
 {"id": "countrylanguage:4", "key": ["ABW", "Spanish"], "value": null},
 {"id": "countrylanguage:5", "key": ["AFG", "Balochi"], "value": null}
 ]
}

+-------------+---------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------------+---------------+------+-----+---------+-------+
| CountryCode | char(3) | NO | PRI | | |
| Language | char(30) | NO | PRI | | |
| IsOfficial | enum('T','F') | NO | | F | |
| Percentage | float(4,1) | NO | | 0.0 | |
+-------------+---------------+------+-----+---------+-------+

## SQL Information ##
sql.connection=jdbc:mysql://192.168.99.19:3306/world
sql.username=root
sql.password=password

## Couchbase Information ##
cb.uris=http://localhost:8091/pools
cb.bucket=default
cb.password=

## Import information
import.tables=ALL
import.createViews=true
import.typefield=type
import.fieldcase=lower

java -cp "./CouchbaseSqlImporter.jar:./mysql-connector-java-5.1.25-bin.jar" com.couchbase.util.SqlImporter import.properties 

Posted by Tug Grall at 10:52 AM 2 comments 👁 Image

Most of the applications have to deal with "master/detail" type of data:

• breweries and beer
• department and employees
• invoices and items 
• ...

• Create a single document for each master and embed all the children in it
• Create a master and child documents and link them using an attribute.

The Data

{
 "type": "dept",
 "id": 10,
 "name": "Accounting",
 "city": "New York"
}

{
 "type": "emp",
 "id": 7782,
 "name": "Blake",
 "job": "Clark",
 "manager": 7839,
 "salary": 2450,
 "dept_id": "dept__10"
}

• dept__10
• emp__20

./cbdocloader -n 127.0.0.1:8091 -u Administrator -p password -b default ~/Downloads/emp-dept.zip 

The View

• the department id extracted from the department document itself or from the employee document depending of the type of document
• an arbitrary number that is used to control the ordering. I put 0 for the department, 1 for the employee
• the name of the department or the employee, this also allows to sort the result by name

The Beer Sample Application

Posted by Tug Grall at 3:38 AM 0 comments 👁 Image

When you are developing a new applications with Couchbase 2.0, you sometimes need to create view dynamically from your code. For example you may need this when you are installing your application, writing some test, or you can also use that when you are building frameworks, and wants to dynamically create views to query data. This post shows how to do it.

Prerequisites

• Couchbase Server 2.0
• Couchbase Jave Client Library 1.1.x
• Beer Sample dataset

Create and Manage Views From Java 

Connect to Couchbase Cluster

import com.couchbase.client.CouchbaseClient;
...
...

 List

Create Views from Java

 function (doc, meta) {
 if(doc.type && doc.type == "beer") {
 emit(doc.name, null);
 }
 }

import com.couchbase.client.protocol.views.DesignDocument;
import com.couchbase.client.protocol.views.ViewDesign;
...
 DesignDocument designDoc = new DesignDocument("dev_beer");

 String viewName = "by_name";
 String mapFunction =
 "function (doc, meta) {\n" +
 " if(doc.type && doc.type == \"beer\") {\n" +
 " emit(doc.name);\n" +
 " }\n" +
 "}";

 ViewDesign viewDesign = new ViewDesign(viewName,mapFunction);
 designDoc.getViews().add(viewDesign);
 client.createDesignDoc( designDoc );
...

import import com.couchbase.client.protocol.views.*;

...
 System.setProperty("viewmode", "development"); // before the connection to Couchbase
...
 View view = client.getView("beer", "by_name");
 Query query = new Query();
 query.setIncludeDocs(true).setLimit(20);
 query.setStale( Stale.FALSE );
 ViewResponse result = client.query(view, query);
 for(ViewRow row : result) {
 row.getDocument(); // deal with the document/data
 }
...

Posted by Tug Grall at 10:42 AM 12 comments 👁 Image

Introduction

In this article I will explain how you can:

1. Install and Configure Memcached on Mac OS X
2. Use Memcached in your Java Application

• A portal is used to aggregate data in a single page. These data could come from different sources : Web Services, Database, ERP, ..... and accessing the data in real time could be costly. So it will be quite interesting to cache the result of the call when possible.
• If the portal is used to aggregate many data from many sources, it is sometime necessary to jump into another application to continue some operation. A distributed and shared cache could be used to manage some context between different applications running in different processes (JVM or even technologies)

$ mkdir memcachedbuild
$ cd memcachebuild

$ curl -O http://www.monkey.org/~provos/libevent-1.4.14-stable.tar.gz
$ tar xzvf libevent-1.4.14-stable.tar.gz
$ cd libevent-1.4.14-stable
$ ./configure
$ make
$ make verify
$ sudo make install  

$ curl -O http://memcached.googlecode.com/files/memcached-1.4.10.tar.gz
$ tar xzvf memcached-1.4.10.tar.gz
$ cd memcached-1.4.10
$ ./configure
$ make
$ make test
$ sudo make install 

$ /usr/local/bin/memcached -d -p 11211

$ telnet 127.0.0.1 11211
Trying 127.0.0.1...
Connected to tgrall-server.
Escape character is '^]'.
set KEY 0 600 16
This is my value
STORED
get KEY
VALUE KEY 0 16
This is my value
END

set <key>  <flags> <expiration_time>  <number_of_bytes> [noreply] \n\n

<value>

• key : the key used to store the data in the cache
  
• flags : a 32 bits unsigned integer that memcached stored with the data
• expiration_time : expiration time in seconds, if you put 0 this means no delay

• number_if_bytes : number of bytes in the data block
• noreply : option to tell the server to not return any value
• value : the value to store and associate to the key.

<repository>
 <id>spy</id>
 <name>Spy Repository</name>
 <layout>default</layout>
 <url>http://files.couchbase.com/maven2/</url>
</repository> 

<dependency>
 <groupid>spy</groupid>
 <artifactid>spymemcached</artifactid>
 <version>2.7.3</version>
</dependency>

package com.grallandco.blog;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.Console;
import java.io.InputStreamReader;
import java.util.Date;
import java.util.logging.Level;
import java.util.logging.Logger;
import net.spy.memcached.AddrUtil;
import net.spy.memcached.MemcachedClient;

public class Test {

 public static void main(String[] args) {
 try {
 
 System.out.print("Enter the new key : ");
 BufferedReader reader = new BufferedReader( new InputStreamReader(System.in));
 String key = null;
 key = reader.readLine();
 
 System.out.print("Enter the new value : ");
 String value = null;
 value = reader.readLine();
 
 MemcachedClient cache = new MemcachedClient(AddrUtil.getAddresses("127.0.0.1:11211"));
 
 // read the object from memory
 System.out.println("Get Object before set :"+ cache.get(key) );

 // set a new object 
 cache.set(key, 0, value );

 System.out.println("Get Object after set :"+ cache.get(key) );
 

 } catch (IOException ex) {
 Logger.getLogger(Test.class.getName()).log(Level.SEVERE, null, ex);
 System.exit(0);
 }

 
 System.exit(0);
 
 }
}

Enter the new key : CITY
Enter the new value : Paris, France
2011-11-16 15:22:09.928 INFO net.spy.memcached.MemcachedConnection: Added {QA sa=/127.0.0.1:11211, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
2011-11-16 15:22:09.932 INFO net.spy.memcached.MemcachedConnection: Connection state changed for sun.nio.ch.SelectionKeyImpl@5b40c281
Get Object before set :null
Get Object after set :Paris, France

get CITY
VALUE CITY 0 13
Paris, France
END

Posted by Tug Grall at 7:12 AM 11 comments 👁 Image

Hey Steve, keep the bean in the Apple!

The news from last week that grabbed the attention of many Java developers was Apple’s announcement of its intentions to deprecate Java in the latest OS X 10.6 update. One sentence stood out in particular, “Developers should not rely on the Apple-supplied Java runtime being present in future versions of Mac OS X,” and raised the question: should Java developers (many of whom, like me, develop on Macs) freak out?

I don’t think so. (Though it prompted additional speculation and follow-on news stories.)

Let’s be realistic. Most applications run on the server side, on Unix/Linux and/or Windows Server – which has nothing to do with Apple or Mac OS X. And more and more applications are running on the cloud, where the language isn’t necessarily irrelevant, but certainly less important than the services that the application exposes. And I’m sure Java will have a big role in ‘development in the cloud,’ as we can already see with Google AppEngine and the VMWare/SpringSource effort.

I think the more interesting question to ask is “Why did Apple do this?”

I believe this is related Apple’s other big news last week: the new “Mac App Store,” which looks like an effort to have one single technology and language to develop “official” applications for Mac. In fact, for all Apple platforms running OS X and iOS, developers should use X Code and Objective C. That’s fine with me, as I enjoy developing small apps for my iPhone and iPad in my spare time, using these tools. But at eXo, many of our developers are using Java, often on Macs, to build our software.

We’re not talking about the same kind of applications. If, in the future, Java does not exist on Macs, it will not cause enterprise developers to abandon Java, but simply force them to move away from their Macs. Personally, I don’t want that to happen. I switched to Mac in 2001, and I’ve been a big fan of all Apple products ever since (most of my extended family are now also on Macs, and they couldn’t care less about Java).

As a Java developer, do I switch back to PC now? Unlikely. I am very confident (overconfident?) that Java will still be present on OS X. The difference is that Apple will simply stop caring about it -- the same way that Microsoft doesn’t care now. I cannot believe that Apple will stop/block Java on their platform. So the future of Java in general, and now on Mac, is fully under the control of the Java community, driven by Oracle and OpenJDK. I am sure we will find many skilled “MacAddicts” to maintain and improve Java on OS X, to at least allow Java developers to run their favorite IDE and test their applications before deploying them on the servers -- keeping the “Write Once, Run Anywhere” a reality (almost...). The only “bad” part is the fact that “Java Desktop” will not borrow any of the cool features of Apple Mac OS X. Not a big deal, since Java Desktop has never been that successful anyway.

So my advice to fellow Java developers is this: if you care, be vocal. Let’s make sure Apple lets the community drive the future of Java on Mac, since the future of the Java platform is still very exciting for many of us.

Original Post on eXo Blog.

Posted by Tug Grall at 11:36 PM 5 comments 👁 Image

When deploying your Web Service client you often need to change the endpoint of the service that  has been set during the code generation. This short post explains how you can set change it at runtime in the client code.

You have two approaches to do that:

• set the endpoint in the Port using the BindingProvider
• get the endpoint URL from the WSDL itself at runtime

Use the Binding Provider to set the endpoint URL

try { 
            EmployeeServiceService service = new EmployeeServiceService();
            EmployeeService port = service.getEmployeeServicePort();

            BindingProvider bp = (BindingProvider)port;
            bp.getRequestContext().put(BindingProvider.ENDPOINT_ADDRESS_PROPERTY, "http://server1.grallandco.com:8282/HumanRessources/EmployeeServiceService");

            Employee emp = port.getEmployee(123);



            System.out.println("Result = "+ emp);
        } catch (Exception ex) {...

           

Use the WSDL to get the endpoint URL

try { 
           EmployeeServiceService service =
           new org.demo.service.EmployeeServiceService
                (new URL("http://server1.grallandco.com:8282/HumanRessources/EmployeeServiceService?wsdl"), 
                new QName("http://service.demo.org/","EmployeeServiceService"));

            EmployeeService port = service.getEmployeeServicePort();
            Employee emp = port.getEmployee(123);

System.out.println("Result = "+ emp);
        } catch (Exception ex) {

Posted by Tug Grall at 3:11 PM 11 comments 👁 Image

I have attended or presented at JavaOne for the last 6 years when I was living in the SF Bay Area... But this year I won't be in San Francisco for JavaOne. As you can guess, I am sad about that... However, I still look at the schedule and events, and here what I would like to do:

Posted by Tug Grall at 10:48 PM 0 comments 👁 Image

With the latest big news around Oracle-BEA and SUN-MySQL deals I have missed an interesting article on Infoq with the following title:
- Request: Sun, Drop Support for JRuby
I have to say that I do agree with Craig Wickesser asking Sun to Drop Support for JRuby.

Posted by Tug Grall at 6:46 AM 3 comments 👁 Image

The new post on the Ruby section of O'Reilly authored by Derek Sivers is quite interesting, starting with the title:

• 7 reasons I switched back to PHP after 2 years on Rails

Posted by Tug Grall at 12:20 AM 5 comments 👁 Image

A friend of mine was asking me how to generate a unique ID for his application... As you probably already know Java SE 5 has introduced the java.util.UUID class to easily generate Universally Unique Identifier (UUID). As usual Wikipedia is a great starting point to learn more about UUID.
Generating the unique ID is as simple as calling the method UUID.randomUUID() in the class. This will give a new instance of UUID that you can now manipulate; for example do a toString() to get the UUID string representation as describe in the specifications; for example 5462dc18-4653-42d1-b4e4-22fc970a6ce5

Resources:

• UUID on Wikipedia
• java.util.UUID Documentation

Posted by Tug Grall at 2:13 AM 3 comments 👁 Image