Tug's Blog

When you are developing a new applications with Couchbase 2.0, you sometimes need to create view dynamically from your code. For example you may need this when you are installing your application, writing some test, or you can also use that when you are building frameworks, and wants to dynamically create views to query data. This post shows how to do it.

Prerequisites

• Couchbase Server 2.0
• Couchbase Jave Client Library 1.1.x
• Beer Sample dataset

Create and Manage Views From Java 

Connect to Couchbase Cluster

import com.couchbase.client.CouchbaseClient;
...
...

 List

Create Views from Java

 function (doc, meta) {
 if(doc.type && doc.type == "beer") {
 emit(doc.name, null);
 }
 }

import com.couchbase.client.protocol.views.DesignDocument;
import com.couchbase.client.protocol.views.ViewDesign;
...
 DesignDocument designDoc = new DesignDocument("dev_beer");

 String viewName = "by_name";
 String mapFunction =
 "function (doc, meta) {\n" +
 " if(doc.type && doc.type == \"beer\") {\n" +
 " emit(doc.name);\n" +
 " }\n" +
 "}";

 ViewDesign viewDesign = new ViewDesign(viewName,mapFunction);
 designDoc.getViews().add(viewDesign);
 client.createDesignDoc( designDoc );
...

import import com.couchbase.client.protocol.views.*;

...
 System.setProperty("viewmode", "development"); // before the connection to Couchbase
...
 View view = client.getView("beer", "by_name");
 Query query = new Query();
 query.setIncludeDocs(true).setLimit(20);
 query.setStale( Stale.FALSE );
 ViewResponse result = client.query(view, query);
 for(ViewRow row : result) {
 row.getDocument(); // deal with the document/data
 }
...

Posted by Tug Grall at 10:42 AM 12 comments 👁 Image

1. Click “Start Menu”
2. Navigate to “All Programs > Accessories”
3. “Right Click” on “Notepad” (or your favorite text editor)
4. Click “Run as Administrator”
   
   👁 Image
   
   
   

1. Click Start Menu
2. Type Services in the Search Program form
3. Click on Services
4. In the Services Application navigate to CouchbaseServer
5. Right Click and Click on Stop 
6. Couchbase is now stopped. 

Posted by Tug Grall at 11:08 AM 6 comments 👁 Image

Yesterday was my last day at eXo... I have been working at eXo since 2008, and we have achieved many exciting things such as building eXo Platform, the open source social platform, and the Cloud-IDE allowing developers to build, test, and deploy applications online.

It was a great experience for me, but it is time for me to jump into a new adventure...

I am joining Couchbase as a Technical Evangelist for the EMEA Region. When I have started to work with NoSQL engines (starting with Google BigTable for Resultri) I really enjoyed the experience and the flexibility that it gives to the developers. Then I choose to work on a documented oriented database because it looks very natural to me, and evaluate the clustering capabilities. This is how I discovered Couchbase and its community.

This new job is a great opportunity for me to do the things that I really like about software:

• Coding applications using different languages and frameworks
• Understanding the sysops/devops related challenges when using a new product/technology
• And finally probably the most important part; sharing it with others!

Posted by Tug Grall at 2:25 AM 3 comments 👁 Image

After some basic articles about Couchbase installation, Node.js integration. Let's now dive into a more complete  example: a chat application.

The first version of the chat should be compliant with the following requirements:

• web based
• single room
• user just needs to enter a login and he can start to interact with other connected users
• user should be able to navigate into the chat history

• Node.js for the application
• with express and socket.io
• Couchbase to persist all the messages

• Using Views that allows you to query and secondary level index and do advanced operation such as sorting, query on key range, ...
• Directly access the data using its key

• First Options: using a view to get the message history
• Second Options: using a counter as a key for the messages

var express = require('express');
var app = express();
var http = require('http');
var server = http.createServer(app);
var io = require('socket.io').listen(server);

var driver = require('couchbase');

driver.connect({
 "username": "",
 "password": "",
 "hostname": "localhost:8091",
 "bucket": "default"}, 
 function(err, couchbase) {
 if (err) {
 throw (err)
 }

 server.listen(8080);

 app.get('/', function(req, res) {
 res.sendfile(__dirname + '/index.html');
 });
...
// Application code
// Socket.io events
...
});

{
 "type": "message",
 "user": "Tug",
 "message": "Hello all !",
 "timestamp": 1349836768909
}

 socket.on('postMessage', function(data) {
 // create a new message
 var message = {
 type: "message",
 user: socket.username,
 message: data,
 timestamp: Date.now()
 }
 var messageKey = message.timestamp +"-"+ message.user;
 io.sockets.emit('updateChatWindow', message);
 couchbase.set(messageKey, JSON.stringify(message),function(err) { }); 
 });

• The postMessage event is called by the client when the user post a new message. 
• A new message object is created with : a type, the user, the message itself and a timestamp.
• The message is sent to the different clients using the io.sockets.emit() function (line 10)
• Finally the message is saved into Couchbase (line 11). As you can see the only thing you have to do is to send the Javascript object as a simple JSON String.

function (doc, meta) { 
 if ( meta.type == "json" && doc.type == "message" ) {
 emit(doc.timestamp, null);
 }
}

{"id":"1352733392477-JOHN","key":1352733392477,"value":null},

curl -X PUT -H 'Content-Type: application/json' http://127.0.0.1:8092/default/_design/chat -d '{"views":{"message_hisory":{"map":"function (doc, meta) {\n \n if ( meta.type == \"json\" && doc.type == \"message\" ) {\n emit(doc.timestamp, null);\n }\n}"}}}'

socket.on('showhistory', function(limit,startkey) {
 limit = (limit == undefined || limit == 0)? 5 : limit;
 var options = {"descending": "true", "limit" : limit, "stale" : "false"};
 if (startkey > 0) {
 options.startkey = startkey-1;
 }
 couchbase.view("chat","message_hisory", options , function(err, resp, view) {
 var rows = view.rows;
 var keys = new Array();
 for( var i = 0; i < rows.length ; i++ ) {
 keys.push( rows[i].id );
 }
 couchbase.get(keys,function(err, doc, meta) {
 socket.emit('updateChatWindow', doc, true);
 });
 });
});

• Use descending order to return the messages from the newest to the oldest
• The number of message to return (limit)
• Ask the view to update the index before returning the rows using the stale=false parameter.
• Use startkey parameter if the client send a specific starting point. 

1. When user connects to the server the system returns the last 5 messages from the database
2. Each time the user posts a message it should be persisted
3. The user can manually load older messages from the database to view the complete chat history

socket.on('postMessage', function(data) {
 // create a new message
 var message = {
 type: "message",
 user: socket.username,
 message: data,
 timestamp: Date.now()
 }
 couchbase.incr("chat:msg_count", function (data, error, key, cas, value ) { 
 var messageKey = "chat:"+ value;
 message.id = value;
 io.sockets.emit('updateChatWindow', message);
 couchbase.set(messageKey, JSON.stringify(message),function(err) { }); 
 });
});

• The value is used to create a new key for the message (line 10)
• The message is push to the different users  (line 12)
• Then the message is saved into Couchbase (line 13)

• a new item that contains the counter, associated to the key : chat:msg_count
• each message will have a key that looks like chat:0, chat:1, chat:2, ... 

socket.on('showHistory', function(limit,startkey) {
 var keys = new Array();
 for (i = startkey; i > (startkey-limit) && i >= 0 ; i--) {
 keys.push("chat:"+i);
 }
 couchbase.get(keys,function(err, doc, meta) {
 socket.emit('updateChatWindow', doc, true);
 });
});

Posted by Tug Grall at 12:45 AM 7 comments 👁 Image

An easy way to create large dataset when playing/demonstrating Couchbase -or any other NoSQL engine- is to inject Twitter feed into your database.

For this small application I am using:

• Couchbase Server 2.0 Server
• Couchbase Java SDK (will be installed by Maven)
• Twitter4J (will be installed by Maven)
• Twitter Streaming API called using Twitter4J

• Consumer key 
• Consumer secret
• Access token
• Access token secret

 ... 
 
 ...

• The mainClass entry allows you to set which class to execute when running java -jar command.
• The Class-Path entry allows you to set the current directory as part of the classpath where the program will search for the twitter4j.properties file.
• The assembly file is also configure to include all the dependencies (Twitter4J, Couchbase client  SDK, ...)

mvn clean package

twitter4j.jsonStoreEnabled=true

oauth.consumerKey=[YOUR CONSUMER KEY]
oauth.consumerSecret=[YOUR CONSUMER SECRET KEY]
oauth.accessToken=[YOUR ACCESS TOKEN]
oauth.accessTokenSecret=[YOUR ACCESS TOKEN SECRET]

couchbase.uri.list=http://127.0.0.1:8091/pools
couchbase.bucket=default
couchbase.password=

jar -jar [path-to-jar]/CouchbaseTwitterInjector.jar

Posted by Tug Grall at 9:11 PM 1 comments 👁 Image

NOTE: The Couchbase Node.js Client Library is currently changing. I will update this article and source code once the API is stable.

Installation

ruby -e "$(curl -fsSkL raw.github.com/mxcl/homebrew/go)"

brew install node

brew install https://github.com/couchbase/homebrew/raw/preview/Library/Formula/libcouchbase.rb

Coding

Create a very simple application

mkdir sample-app
cd sample-app

var http = require("http");

function onRequest(request, response) {
 response.writeHead(200, {"Content-Type": "text/plain"});
 response.write("Hello World");
 response.end();
}

var server = http.createServer(onRequest);
server.listen(8080);

console.log("> SERVER STARTED");

node app.js

Install and use Couchbase client for node.js

npm install couchbase

var http = require("http");

// load the Couchbase driver and connect to the cluster
var driver = require('couchbase');
var cb = new driver.Couchbase("localhost:8091", null, null, "default");

...
...

function insertData() {
 // insert employees in Couchbase
 var emps = [{
 "type": "employee",
 "id": 100,
 "name": "Thomas",
 "dept": "Sales",
 "salary": 5000
 }, {
 "type": "employee",
 "id": 200,
 "name": "John",
 "dept": "Development",
 "salary": 4500
 }, {
 "type": "employee",
 "id": 300,
 "name": "Jane",
 "dept": "Marketing",
 "salary": 5000
 }]

 // Insert the data in Couchbase using the add method ()
 for (index = 0; index < emps.length; index++) {
 cb.add(JSON.stringify(emps[index].id), JSON.stringify(emps[index]), 0, undefined, function(data, err, key, cas) {
 if (err && err != 12) { // 12 : LCB_KEY_EEXISTS 
 console.log("Failed to store object:\n" + err);
 }
 });
 }
}

server.listen(8080, insertData());

var http = require("http");

// load the Couchbase driver and connect to the cluster
var driver = require('couchbase');
var cb = new driver.Couchbase("localhost:8091", null, null, "default");
// load the baseview module
var baseview = require('baseview')({url: 'http://localhost:8092', bucket: 'default'});


...
...

function insertData() {

 //create a new view
 baseview.setDesign('design_employees', {
 'basic_list': {
 'map': "function (doc, meta) { if(doc.type == 'employee') {emit(meta.id, doc.name);}}"
 }
 },
 function(err, res){
 if (err != null) console.log(err);
 }
 ); 

...

}
...
 

function onRequest(request, response) {
 response.writeHead(200, {
 "Content-Type": "text/plain"
 });
 response.write("See list of employees in the console");
 var params = 
 { 'descending' : 'true'
 , 'include_docs' : 'true'
 };
 baseview.view('design_employees', 'basic_list', params, function(error, data) {
 for( var i = data.rows.length-1,length = data.rows.length ; i >= 0; i-- ) {
 var employee = data.rows[i].doc.json;
 console.log(employee); 
 } 
 }); 
 response.end();
}

var http = require("http");
var driver = require('couchbase');
var cb = new driver.Couchbase("localhost:8091", null, null, "default");
var baseview = require('baseview')({url: 'http://127.0.0.1:8092',bucket: 'default'});

function onRequest(request, response) {
 response.writeHead(200, {
 "Content-Type": "text/plain"
 });
 response.write("See list of employees in the console");
 var params = {
 'descending': 'true',
 'include_docs': 'true'
 };
 baseview.view('design_employees', 'basic_list', params, function(error, data) {
 for (var i = data.rows.length - 1, length = data.rows.length; i >= 0; i--) {
 var employee = data.rows[i].doc.json;
 console.log(employee);
 }
 });
 response.end();
}

function insertData() {
 //create a new view
 baseview.setDesign('design_employees', {
 'basic_list': {
 'map': "function (doc, meta) { if(doc.type == 'employee') {emit(meta.id, doc.name);}}"
 }
 }, function(err, res) {
 if (err != null) console.log(err);
 });

 // insert employees in Couchbase
 var emps = [{
 "type": "employee",
 "id": 100,
 "name": "Thomas",
 "dept": "Sales",
 "salary": 5000
 }, {
 "type": "employee",
 "id": 200,
 "name": "John",
 "dept": "Development",
 "salary": 4500
 }, {
 "type": "employee",
 "id": 300,
 "name": "Jane",
 "dept": "Marketing",
 "salary": 5000
 }]


 // Insert the data in Couchbase using the add method ()
 for (index = 0; index < emps.length; index++) {
 cb.add(JSON.stringify(emps[index].id), JSON.stringify(emps[index]), 0, undefined, function(data, err, key, cas) {
 if (err && err != 12) { // 12 : LCB_KEY_EEXISTS 
 console.log("Failed to store object:\n" + err);
 }
 });
 }
}

var server = http.createServer(onRequest);

server.listen(8080, insertData());

console.log("> SERVER STARTED");

Posted by Tug Grall at 7:13 PM 9 comments 👁 Image

Introduction

1. Install Couchbase 
2. Create Data 
3. Query Data 

• Insert some data and query them

Insert Data

1. Click on Documents button
2. Click on Create Document
3. Since each document must have an id for example 100.
4. Couchbase save the document and add some metadata such as _rev, $flags, expiration
5. Add new attributes to the document that describe an employee : Name, Departement and Salary, then save it. You just need to update the JSON object with values
   {
     "_id": "100",
     "name": "Thomas",
     "dept": "Sales",
     "salary": 5000
   }

200,Jason,Technology,5500
300,Mayla,Technology,7000
400,Nisha,Marketing,9500
500,Randy,Technology,6000
501,Ritu,Accounting,5400

Query Data

Access document directly from its ID

• http://127.0.0.1:8092/default/300  

• 8092 is the Couch API REST port used to access data (where 8091 is the port for the Admin console)
• default is the bucket in which the document is stored
• 300 is the id of the document

Search your data with queries

•  "Give me all the employee of the Technology department"

1. Click on the Views tab (be sure you are on the default bucket)
2. Click on the "Create Development View"
3. Enter the Document and View name:
• Document Name : _design/dev_dept
• View Name : dept
4. Cick Save
5. Click on your View to edit it

function (doc) {
 emit(doc.dept, null);
}

{"total_rows":6,"rows":[
 {"id":"501","key":"Accounting","value":null},
 {"id":"400","key":"Marketing","value":null},
 {"id":"100","key":"Sales","value":null},
 {"id":"200","key":"Technology","value":null},
 {"id":"300","key":"Technology","value":null},
 {"id":"500","key":"Technology","value":null}
]
}

• http://127.0.0.1:8092/default/_design/dev_dept/_view/dept

• http://127.0.0.1:8092/default/_design/dev_dept/_view/dept?key="Technology"

{"total_rows":6,"rows":[
 {"id":"200","key":"Technology","value":null},
 {"id":"300","key":"Technology","value":null},
 {"id":"500","key":"Technology","value":null}
]
}

• http://127.0.0.1:8092/default/_design/dev_dept/_view/dept?key="Technology"&include_docs=true

{"total_rows":6,"rows":[
 {"id":"200","key":"Technology","value":null,"doc": {"_id":"200","_rev":"1-1de6e6751608eada0000003200000000","$flags":0,"$expiration":0,"name":"Jason","dept":"Technology","salary":5500}},
 {"id":"300","key":"Technology","value":null,"doc":{"_id":"300","_rev":"1-f3e44cee742bfae10000003200000000","$flags":0,"$expiration":0,"name":"Mayla","dept":"Technology","salary":7000}},
 {"id":"500","key":"Technology","value":null,"doc": {"_id":"500","_rev":"1-05780359aac8f3790000003200000000","$flags":0,"$expiration":0,"name":"Randy","dept":"Technology","salary":6000}}
]
}

function (doc) {
 emit(doc.salary, null);
}

• http://127.0.0.1:8092/default/_design/dev_salary/_view/salary?startkey=5000&endkey=6000&include_docs=true 

{"total_rows":6,"rows":[
 {"id":"100","key":5000,"value":null,"doc":{"_id":"100","_rev":"1-0f33580d780014060000002e00000000","$flags":0,"$expiration":0,"name":"Thomas","dept":"Sales","salary":5000}},
 {"id":"501","key":5400,"value":null,"doc":{"_id":"501","_rev":"1-b1fe5bc79637720e0000003100000000","$flags":0,"$expiration":0,"name":"Ritu","dept":"Accounting","salary":5400}},
 {"id":"200","key":5500,"value":null,"doc":{"_id":"200","_rev":"1-1de6e6751608eada0000003200000000","$flags":0,"$expiration":0,"name":"Jason","dept":"Technology","salary":5500}},
 {"id":"500","key":6000,"value":null,"doc":{"_id":"500","_rev":"1-05780359aac8f3790000003200000000","$flags":0,"$expiration":0,"name":"Randy","dept":"Technology","salary":6000}}
]
}

Conclusion

• Install Couchbase
• Create data using the Admin Console
• Query data with views

Posted by Tug Grall at 8:31 AM 7 comments 👁 Image

In this screencast I explain the support of multi-lingual content of eXo Platform 3.5. The different features that you can see in this video are:

• Support of multiple languages from URL 
• Configure the eXo File Explorer to add support for multi-lingual content (new button)
• Content translation
• Add new language to the platform

Posted by Tug Grall at 9:11 AM 0 comments 👁 Image

Like many developers I am using Twitter Boostrap for my Web applications. Using this framework has been very helpful for me, since I am really not a good HTML/CSS developer. For now, on my site Resultri I am using the default look and feel, will customize it later.

Lately, I wanted to integrate Google Map to my application, and when testing it, I had the bad surprise to see that the Controls and WindowInfo are not printed correctly as you can see in the screen shot below:

• override the style of the img tag for the div that contains your map.

Posted by Tug Grall at 1:00 PM 4 comments 👁 Image

Last week I was invited to present eXo Cloud IDE during the SAP Cloud Inside. This SAP Community event was a great opportunity to discuss about the cloud with an interesting point of view: the impact of the cloud for SAP customers (especially administrators and developers).

During this presentation I have introduced the eXo Cloud IDE, and I did a demonstration in which O have built and deployed applications : Open Social Gadgets, Ruby on Rails and Java/Spring, and explain how it could be extended to SAP business services.

Here the slides that I have used during this presentation:

Posted by Tug Grall at 11:56 PM 0 comments 👁 Image

Introduction

Many Google AppEngine developers have been waiting for the Full Text Search feature, especially coming from Google the biggest search engine on the Web. I was quite happy to see that Google team is working on it as you can check in the Google I/O 2011 session : Full Text Search by Bo Majewski, Ged Ellis . As far as I know the very promising indexing service is not yet available.

In this article I will explain how you can provide some kind of full text search in your application using services available App Engine services.

In my specific use case I do not ask for a lot of feature, I just need to have simple search a string in various attributes of my entities independently of the case, and possible special characters (such as è,é, ... ). I am far of being an expert of Google Datastore API but I did not find any simple way to achieve this directly using the Java API. What I have done to solve this issue is to duplicate a part of my data into the Google Cloud SQL to use the MySQL fulltext search capabilities.

Prerequisites
To achieve the following tasks you need to :

• Install the Google AppEngine SDK for Java 
• Install MySQL 5.5
• A Google Account with AppEngine and Cloud SQL services.
• Optionally : Eclipse Plugin to develop, debug and deploy my Web application to GAE

• the full application code : https://github.com/tgrall/gae-full-text-search
• take a look of the running application : http://gae-fulltext-search.appspot.com/

import com.google.appengine.api.datastore.DatastoreService;
import com.google.appengine.api.datastore.DatastoreServiceFactory;
import com.google.appengine.api.datastore.Entity;

//...
//...

 Entity article = new Entity("Article");
 article.setProperty("title", "MySQL Tutorial");
 article.setProperty("body", "DBMS stands for DataBase ...");
 datastore.put(article);

 article = new Entity("Article");
 article.setProperty("title", "Datastore Index Selection and Advanced Search");
 article.setProperty("body", "Learn how recent improvements to the query planner ... function in your application");
 datastore.put(article); 

• I store the text values on which I want to do some search in Google Cloud SQL and use the Full Text features of MySQL.

CREATE SCHEMA search_values DEFAULT CHARACTER SET utf8 ;

USE search_values;


CREATE TABLE articles (
 entity_key varchar(250),
 title text,
 body text,
 PRIMARY KEY RESULTS_PK (entity_key),
 FULLTEXT (title,body)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;

1. Copy the MySQL JDBC driver into your Google App Engine SDK directory, under /lib/impl/. You can download the MySQL JDBC driver here.
2. In Eclipse, select your Java package.
3. Click Run > Run Configurations.
4. Expand the Web Application menu item.
5. Add the following lines into the VM Arguments pane:
   

   -Drdbms.server=local
   -Drdbms.driver=com.mysql.jdbc.Driver
   -Drdbms.url=jdbc:mysql://localhost:3306/search_values?user=username&password=password
   

6. Click the Classpath tab.
7. Select your project and click Add External JARs...
8. Navigate to the Google App Engine SDK directory, then lib/impl, and select the JDBC driver JAR file. Click Open. The driver JAR is listed under User Entries.
9. Click Apply.

 Connection conn = null;
 try {
 DriverManager.registerDriver(new AppEngineDriver());
 conn = DriverManager.getConnection("jdbc:google:rdbms://[your db instance]/search_values");
 conn.setAutoCommit(false); 
 String statement = "REPLACE INTO articles (entity_key, title, body) VALUES( ? , ? , ? )";
 PreparedStatement stmt = conn.prepareStatement(statement);

 DatastoreService datastore = DatastoreServiceFactory.getDatastoreService();
 Query q = new Query("Article"); 
 PreparedQuery pq = datastore.prepare(q);

 // loop on each entity and insert the values in the SQL Table
 for (Entity result : pq.asIterable()) {
 stmt.setString(1, KeyFactory.keyToString(result.getKey()) );
 stmt.setString(2, result.getProperty("title").toString() );
 stmt.setString(3, result.getProperty("body").toString() );
 stmt.executeUpdate();
 conn.commit();
 }



 } catch (SQLException e) {
 e.printStackTrace();
 } finally {
 if (conn != null)
 try {
 conn.close();
 } catch (SQLException ignore) {}
 }

• I manage the connection directly in my code (I have not looked yet if I can use datasources/connection pool in the context of Google AppEngine)
• Line #3: registering the AppEngine driver that is responsible of managing the connection, expecially work in development -local MySQL- or production mode -CloudSQL-.
• Line #4 : Get the connection. It is interesting to mention that in development the connection URL is grabbed from the environment variable Drdbms.url you have set previously. We will see later how we move this to the cloud. This is the magical part of the AppEngineDriver that manages different connection types Local MySQL or CloudSQL depending of the context
• After these lines, the code is quite simple :
• Get all the Articles entities from the datastore and loop
• "Upsert" the database record (REPLACE INTO syntax)
• Line #15 is storing the Key of the entity in a safe string using the KeyFactory.keyToString() method.

• public Iterable

1. Execute a SQL query
2. For each result, get the Entity using the Key
3. Return the list of Entities

 public Iterable

• "database" : the method returns all the articles concerning database, mysql, RDBMS independently of the case.
• "index" " the method returns all the articles talking about indexing/indexes or search.

Posted by Tug Grall at 6:47 AM 0 comments 👁 Image