Agajorte: Java, Linux e Gastronomia

NoSQL: onde, como e por quê? Cassandra e MongoDB

2011-11-10T06:12:00.000-08:00

Segue material da apresentação que fiz no dia 10/11 no Seminário de Gerenciamento de Dados em Software Livre no SERPRO.

View more presentations from Rodrigo Hjort.

Visão geral sobre bancos de dados NoSQL e detalhes técnicos dos modelos de dados e arquiteturas das implementações Apache Cassandra e MongoDB.

A Linux trick on Oracle setup

2011-10-19T08:22:00.000-07:00

Have you ever tried to install Oracle Database on GNU/Linux and faced the insufficient memory issue below?

Oracle Database 10g Express Edition requires 1024 MB of swap space. This system has ??? MB of swap space.

Well, I've just been caught by that when attempting to set it up on a virtual machine.


# dpkg -i oracle-xe-universal_10.2.0.1-1.1_i386.deb
(Reading database ... 125364 files and directories currently installed.)
Unpacking oracle-xe-universal (from oracle-xe-universal_10.2.0.1-1.1_i386.deb) ...
This system does not meet the minimum requirements for swap space.  Based on 
the amount of physical memory available on the system, Oracle Database 10g 
Express Edition requires 1024 MB of swap space. This system has 1019 MB 
of swap space.  Configure more swap space on the system and retry the installation.
dpkg: error processing oracle-xe-universal_10.2.0.1-1.1_i386.deb (--install):
 subprocess new pre-installation script returned error exit status 1
Errors were encountered while processing:
 oracle-xe-universal_10.2.0.1-1.1_i386.deb

Fortunately a true Operating System can handle it seamlessly! Here are the steps I followed.

Setup is claiming about just 5 megabytes... Well, let's give him it. But no more and no less! Let's create a zeroed dummy file with dd:


# dd if=/dev/zero of=5m.swp bs=1M count=5
5+0 records in
5+0 records out
5242880 bytes (5.2 MB) copied, 0.0132488 s, 396 MB/s

Here you go, an exactly 5 MB size zeroed file:


# ls -la 5m.swp 
-rw-r--r-- 1 root root 5242880 2011-10-19 10:39 5m.swp

Let's check out the available memory in the system using free:


# free
             total       used       free     shared    buffers     cached
Mem:       1026080     921612     104468          0      27036     674936
-/+ buffers/cache:     219640     806440
Swap:      1046524       2552    1043972

Alright, you need it, we'll give you:


# swapon 5m.swp

Are you satisfied now?


# free
             total       used       free     shared    buffers     cached
Mem:       1026080     921736     104344          0      27044     674936
-/+ buffers/cache:     219756     806324
Swap:      1051640       2552    1049088

Let's retry running Oracle installer:


# dpkg -i oracle-xe-universal_10.2.0.1-1.1_i386.deb
(Reading database ... 125364 files and directories currently installed.)
Unpacking oracle-xe-universal (from oracle-xe-universal_10.2.0.1-1.1_i386.deb) ...

That's it! Although that ridiculous additional 5 MB swap won't be automatically available for the next reboot, we still can tune Oracle to allocate less memory than standard settings.

Générer des fichiers checksum pour Maven

2011-10-13T11:07:00.000-07:00

Dans un répertoire Maven v2 les fichiers JAR ont l'intégrité contrôlée par des fichiers contenant des sommes MD5 et SHA1. En général ces fichiers auxiliaires sont appelés comme le nom du fichier JAR accompagnés des extensions ".md5" ou ".sha1". Par exemple, pour la librairie "jcommon-0.9.6.jar" il doit exister "jcommon-0.9.6.jar.md5" et "jcommon-0.9.6.jar.sha1".

Ces ".md5" et ".sha1" ne sont que des fichiers texte avec une chaîne string correspondant à la somme de contrôle calculée à partir de l'archive. Cette somme représente une signature unique pour chaque fichier et peut garantir qu'il ne soit pas craqué ou corrompu.

Pour générer ces fichiers au moment de la livraison d'une librairie Java, ils existent des plugins Maven pour les créer automatiquement. Néanmoins, on peut avoir le cas où des librairies JAR n'aient pas ses fichiers de contrôle (par exemple, si on a créé cette partie du répo manuellement). Cette façon, lors d'une résolution de dépendance au Maven, on va avoir des notifications telles qu'au-dessous :

[WARNING] *** CHECKSUM FAILED - Checksum failed on download

Pour illustrer ce soucis, voici un exemple d'un arbre partiel au Maven :


|-- jfree-former
|   |-- jcommon
|   |   `-- 0.9.6
|   |       |-- jcommon-0.9.6.jar
|   |       `-- jcommon-0.9.6.pom
|   `-- jfreechart
|       `-- 0.9.21
|           |-- jfreechart-0.9.21.jar
|           `-- jfreechart-0.9.21.pom

Il faut d'abord créer un script Shell appelée checksum.sh avec le contenu ci-dessous :


#!/bin/bash

if [ $# -ne 2 ]
then
	echo "Usage : checksum [md5|sha1] <file-name>"
	echo "Sample: checksum sha1 /tmp/dir/myfile.jar"
	exit 1
fi

format=$1
file="$2"

if [ ! -f $file ]
then
	echo "File not found: $file"
	exit 2
fi

if [ "$format" == "md5" -o "$format" == "sha1" ]
then
	${format}sum "$file" | cut -d' ' -f1 | tr -d "\n" > "$file.$format"
else
	echo "Please choose a format: md5 or sha1"
	exit 2
fi

echo "Created checksum file $file.$format"

Ensuite, on doit créer un autre script nommé generate-checksums.sh contenant ces lignes :


#!/bin/bash

EXTENSIONS="jar pom"
ALGORITHMS="sha1 md5"

PROGDIR=`dirname $0`
export PATH="$PATH:$PROGDIR"

for ext in $EXTENSIONS
do
	for alg in $ALGORITHMS
	do
		find -type f -name "*.$ext" -exec checksum.sh $alg {} \;
	done
done

N'oubliez pas de donner des permissions d'exécution à ses fichiers .sh, en roulant le commande chmod -x *.sh.

Maintenant, il faut seulement aller au répertoire désiré et ensuite exécuter le script generate-checksums.sh. Voyez :


$ cd /var/maven/repo/

$ /home/user/scripts/generate-checksums.sh
Created checksum file ./jfree-former/jcommon/0.9.6/jcommon-0.9.6.jar.sha1
Created checksum file ./jfree-former/jfreechart/0.9.21/jfreechart-0.9.21.jar.sha1
Created checksum file ./jfree-former/jcommon/0.9.6/jcommon-0.9.6.jar.md5
Created checksum file ./jfree-former/jfreechart/0.9.21/jfreechart-0.9.21.jar.md5
Created checksum file ./jfree-former/jcommon/0.9.6/jcommon-0.9.6.pom.sha1
Created checksum file ./jfree-former/jfreechart/0.9.21/jfreechart-0.9.21.pom.sha1
Created checksum file ./jfree-former/jcommon/0.9.6/jcommon-0.9.6.pom.md5
Created checksum file ./jfree-former/jfreechart/0.9.21/jfreechart-0.9.21.pom.md5

Et voici le résultat pour le cas d'exemple :


|-- jfree-former
|   |-- jcommon
|   |   `-- 0.9.6
|   |       |-- jcommon-0.9.6.jar
|   |       |-- jcommon-0.9.6.jar.md5
|   |       |-- jcommon-0.9.6.jar.sha1
|   |       |-- jcommon-0.9.6.pom
|   |       |-- jcommon-0.9.6.pom.md5
|   |       `-- jcommon-0.9.6.pom.sha1
|   `-- jfreechart
|       `-- 0.9.21
|           |-- jfreechart-0.9.21.jar
|           |-- jfreechart-0.9.21.jar.md5
|           |-- jfreechart-0.9.21.jar.sha1
|           |-- jfreechart-0.9.21.pom
|           |-- jfreechart-0.9.21.pom.md5
|           `-- jfreechart-0.9.21.pom.sha1

Voilà ! Désormais votre répertoire Maven contient des informations de vérification pour les fichiers et la résolution de dépendances ne lancera plus le message "CHECKSUM FAILED".

Si vous voulez, le code source complet peut être obtenu dans cette adresse : https://gitorious.org/shell-scripts/checksum

Mantendo atualizado o índice FTS no PostgreSQL

2011-10-04T07:32:00.000-07:00

Dando sequência ao post anterior "Full Text Search em português no PostgreSQL", veremos como manter atualizado o índice de busca textual, particularmente no caso de os dados da tabela sofrerem muitas atualizações (i.e., inclusões e modificações).

Só para relembrar, o diagrama a seguir ilustra o funcionamento do mecanismo de Full Text Search no PostgreSQL, em que fazem parte processos como parser, normalizador e indexador:

Para exemplificar o problema apresentado no início desse post, considere que a tabela MUNICIPIOS esteja devidamente populada, tendo a coluna de suporte para a busca textual criada e o índice do tipo GIN (ou GiST) associado. Sendo assim, podemos fazer consultas SQL desse tipo:

SELECT nome, uf FROM municipios
WHERE busca @@ plainto_tsquery(simples('agua lindoia'));

Veja o resultado:

curso=# SELECT nome, uf FROM municipios
curso-# WHERE busca @@ plainto_tsquery(simples('agua lindoia'))
       nome       | uf 
------------------+----
 Águas de Lindóia | SP
(1 registro)

O que acontece agora se incluirmos um novo registro na tabela? Veja:

INSERT INTO municipios (codigo, nome, uf)
VALUES (100, 'Águas de Lindóia do Sul', 'RS');

Ao executarmos a busca anterior, o município recém-incluído não aparecerá... Isso porque a coluna de suporte à busca não foi atualizada. Veja o conteúdo dela:

curso=# SELECT nome, uf, busca FROM municipios WHERE codigo = 100;
          nome           | uf | busca 
-------------------------+----+-------
 Águas de Lindóia do Sul | RS | 
(1 registro)

Para manter essa coluna atualizada automaticamente em operações de INSERT ou UPDATE, precisamos de um disparador (trigger). No PostgreSQL criamos uma função em linguagem procedural (geralmente em PL/pgSQL) e em seguida a associamos a um disparador.

Sendo assim, crie a função de trigger com o código abaixo:

CREATE FUNCTION municipios_trigger()
RETURNS trigger AS $$
begin
  new.busca := to_tsvector(simples(new.nome));
  return new;
end
$$ LANGUAGE plpgsql;

Ao ser chamada, ela fará com que o conteúdo da coluna "busca" seja preenchido com o texto gerado da normalização da coluna "nome".

Em seguida, crie o disparador de atualização:

CREATE TRIGGER municipios_tsupdate
BEFORE INSERT OR UPDATE ON municipios
FOR EACH ROW EXECUTE PROCEDURE municipios_trigger();

Ou seja, antes de cada INSERT ou UPDATE na tabela, a função municipios_trigger() será invocada para preencher automaticamente a coluna "busca".

Agora experimente modificar aquele registro inserido acima:

UPDATE municipios SET uf = uf WHERE codigo = 100;

Observe como ficou o conteúdo daquela linha:

curso=# SELECT nome, uf, busca FROM municipios WHERE codigo = 100;
          nome           | uf |           busca           
-------------------------+----+---------------------------
 Águas de Lindóia do Sul | RS | 'agu':1 'lindo':3 'sul':5
(1 registro)

Finalmente efetue a busca textual que havia falhado:

curso=# SELECT nome, uf FROM municipios
curso-# WHERE busca @@ plainto_tsquery(simples('agua lindoia'));
          nome           | uf 
-------------------------+----
 Águas de Lindóia do Sul | RS
 Águas de Lindóia        | SP
(2 registros)

Tente agora fazer uma modificação no nome desse município:

UPDATE municipios SET nome = 'Águas Quentes do Sul' WHERE codigo = 100;

E então refaça a busca considerando o novo nome:

curso=# SELECT codigo, nome, uf FROM municipios
WHERE busca @@ plainto_tsquery(simples('agua quente'));
 codigo |         nome         | uf 
--------+----------------------+----
    100 | Águas Quentes do Sul | RS
(1 registro)

Índice sempre up-to-date! Muito fácil, né? :D

Referências

[1] PostgreSQL 8.3 Documentation - Full Text Search

Transferring photos to Nintendo 3DS

2011-09-08T19:59:00.000-07:00

Yesterday my daughter was playing with her Nintendo 3DS console and then asked me to put in there some pictures I had in the computer.

I thought that should be fairly easy, as the system is provided with a regular SD card and I already copied there some MP3 files which worked successfully.

Well, after connecting the SD card into my laptop, I just started browsing its folders and quickly found some named with the pattern "199NIN03" inside DCIM. They were there: files with JPG and MPO extensions. The latter is actually a twice-JPG, the so-called "3D picture" taken with the console's dual camera.

My first attempt was to create a new directory with the name I'd like and start copying JPG files in there. Unmounted the drive in Linux, put it on the console, start it and go to its photo browser application. Nothing!

Yes, I had previously read the f*cking manual [RTFM], but I've found nothing about transferring pictures from a computer to Nintendo 3DS. Indeed the way back is possible (i.e., to copy files from the console to a PC).

What do we do in these cases? We start gooooogleing! \o/

There were not so much entries, but this post was bulls-eye: http://techforums.nintendo.com/message/33675

"I got my N3DS the day after its release, and I've been enjoying it except there's nothing telling me how to put pictures from a PC to the SD card. I looked around and only found a guide on putting music on my SD card. Help appreciated."

Okey-dokey! That guy is sharing my sufferings. Look what the "expert" user answered:

"Sorry, but there isn't an easy way to transfer pictures from a PC to the 3DS, as the pictures would need to be in the exact same format, size and have all the information on them as if the 3DS had actually taken the picture. And it is the same as with the DSi and XL, so really the only easy work around would be to view the picture on a computer monitor and take its picture with the 3DS. ;)"

OMG, I didn't believe what I've read. To take pictures of the computer screen with Nintendo 3DS? It just can't be true, what a noob's advice!

Now that sounded like challenge for me! :D

My next step was to analyze the files taken with 3DS, as they were perfectly visible in its browsing tool. That's what I realized from the files:

1. they are named with the pattern "HNI_9999.JPG" and "HNI_9999.MPO" (when taken "in 3D")
2. their resolution was 640x480
3. their average size was 50kB
4. they had several fields in JPEG's header: camera make and model, and date/time taken
5. file command gave this output:HNI_0018.JPG: JPEG image data, EXIF standard 2.2, baseline, precision 0, 4360x480

Then I started hacking around with these f*cking procedures: plug SD card into PC, analyze and copy files, remove SD, plug it into Nintendo 3DS, and... nothing appeared! First I tried renaming folders and filenames according to the patterns the console was expecting. Second I tried to rescale the pictures to 640x480, by using ImageMagick (see my other post):

$ convert -scale 640x480 -quality 85 source/HNI_0018.JPG destin/HNI_0018.JPG

Yet nothing! I wondered the issue was with the JPEG's header. Then I asked Debian's APT for some magical tool designed to handle those cr*ppy header fields... Among several options, I chose jhead. Indeed I found jhead wonderful, so that I could easily manipulate a JPEG's EXIF header from the Linux shell.

When issuing jhead onto the original 3DS file, I had this:

$ jhead HNI_0018.JPG
File name : HNI_0018.JPG
File size : 49610 bytes
File date : 2011:09:08 20:20:09
Camera make : Nintendo
Camera model : Nintendo 3DS
Date/Time : 2011:09:05 14:31:20
Resolution : 640 x 480

So I found an interesting option on jhead: it can copy a JPEG's file header into another file! That's what I did:

$ jhead -te HNI_0018.JPG HNI_0030.JPG

Well, now at least my desired JPEG file had the proper and expected header, right? F*cking cycle again and... nothing! The console was still not recognizing my picture files copied from a PC.

I thought it could be an issue of duplicated timestamps, so I tried to change the file's timestamp (through touch command) and the date/time in JPEG's header (via jhead). No way!

That expert guy in the forum was definitely wrong: even with the proper naming and file formats, the console still does not accept pictures from the outside world! That lock-in stinks like some other company's behavior. Then I turned Nintendo 3DS over looking for some fruit logo on its back! :P

Deep breath. I realized that 3DS was able to save pictures from the Internet browser that comes with the system. Hmmm, that should be a hint! When observing JPEG files taken from Internet, I discovered that they didn't bear those blessed EXIF headers... :P (Well, at least I learned about jhead tool.)

What if I put my own picture files in the Internet and try to access the respective HTML page from 3DS? That's what I did: raised my apache2 service on Linux and pointed the browser in 3DS to my local URL address. After displaying the image, I was able to save it locally in the console! OK, that should be a solution.

Indeed the JPEG was successfully persisted in the proper folder with its preferred name. However, the original file format and EXIM header were kept (I checked it using Linux diff command).

The task of choosing which pictures to import into 3DS I gave to my daughter, but I still needed to make it easier for her. So, in order to display thumbnails rather than Apache's default file listing, I created a small Shell Script:

$ for a in *.JPG *.jpg; do echo "<a href='$a'><img src='$a' border=5 height='25%'/></a><br/>"; done > fotos.html

This was to be run inside each folder containing candidate pictures to be saved into 3DS. In Apache HTTPD's /var/www/ directory I created symbolic links to those directories.

After checking the availability of my local pages from 3DS, I finally returned this gadget to my daughter. Then I taught her how to browse the pages and save the pictures she wanted. She was quite happy for that besides the extended effort.

Wooh, what a lotta work! Life could be easier, isn't it Nintendo?

Introdução ao MongoDB

2011-08-13T21:42:00.000-07:00

Aplicações modernas provaram que bancos de dados do tipo NoSQL são inevitáveis para o sucesso e continuidade de empresas altamente dependentes da Internet. Vide exemplos como Google, Yahoo, Amazon, Twitter e Facebook.

Todavia, existem inúmeras soluções disponíveis e nenhum padrão sobre como manipular, trafegar ou consultar as informações contidas nos bancos NoSQL. Mesmo a classificação (ou melhor dizer, a taxonomia) dessa zoologia de novos bancos ainda está (perdoem-me o trocadilho!) nebulosa... Quem sabe futuramente tenhamos um ANSI-NoSQL...

O fato é que algumas dessas tecnologias provaram ser apenas estufas para o meio acadêmico, enquanto que outras chegaram a evoluir a ponto de serem aceitas por empresas que apostam no pioneirismo. Uma dessas tecnologias de sucesso foi o MongoDB.

View more presentations from hjort.

Nesta apresentação são introduzidos conceitos como as Grandes Rupturas (IMS x RDBMS x NoSQL), o que é o MongoDB, o Modelo de Dados Orientado a Documentos, JSON e BSON, tipos de dados no MongoDB, operações (Insert, Update, Delete), Modificadores Atômicos, Linguagem de Consulta, Indexação, Agregação e Map/Reduce, Capped Collections, GridFS, Server-Side Scripting, Replicação (Master/Slave e Replica Sets), Arquitetura com Sharding, Auto-Sharding + Replicação e outras tecnologias e detalhes envolvidos no banco de dados MongoDB.

Easily creating thumbnails on Linux

2011-08-08T11:39:00.000-07:00

Have you ever needed to scale down a bunch of picture files to a smaller resolution?

If you used a GUI-based program such as gimp or Photoshop, and spent a reasonable time on that task, you'll surely appreciate this post. The same applies if you're not a slave on mice and thus prefer to open command-line terminals rather than double-clicks.

On Linux there is ImageMagick, a software suite that lets you create, edit, and compose bitmap images. It can read, convert and write images in a variety of formats including GIF, JPEG, PNG, TIFF, etc. We can use it to translate, flip, mirror, rotate, scale, shear and transform images, adjust image colors, apply various special effects, or draw text, lines, polygons, ellipses and Bezier curves.

The most important feature to consider here is that ImageMagick includes a number of command-line utilities for manipulating images. That is, text-mode, no GUI needed for editing your images.

We say that ImageMagick can modify or create images automagically and dynamically!

For instance, to rescale (to 800 by 600 pixels) and lower the quality (by 70%) of a JPEG file, the following instruction can be used:

$ convert -scale 800x600 -quality 70 before.jpg after.jpg

Thus, suppose you have a directory hierarchy named source filled with your pictures and you need to create a similar structure but with thumbnails instead. The resulting directory will be called destin.

First of all, open a terminal. :D Then, change to source directory and create the new destin (in this example they're parallel):

$ cd source

$ mkdir ../destin

Second, create subdirectories on destin following the existing source structure:

$ find -type d -exec mkdir -p "../destin/{}" \;

If you wish, issue a tree or simple find command on destin just to check its contents.

At last, run find command along with convert in order to create thumbnails in batch mode:

$ find -type f -exec convert -scale 1024x768 -quality 85 "{}" "../destin/{}" \;

That's it, Power to the Shell! And a Happy 20th Anniversary, Linux!

Full Text Search em português no PostgreSQL

2011-07-29T13:51:00.000-07:00

Full Text Searching (ou simplesmente busca textual ou FTS) é uma poderosa ferramenta disponível em bancos de dados que visa aumentar a probabilidade de sucesso nas consultas efetuadas pelos usuários.

Operadores de busca textual existem nos SGBDs há anos. O PostgreSQL oferece, além do LIKE, o ILIKE (versão que desconsidera a capitalização) e operadores baseados em expressão regular (i.e., ~ e ~*) para tipos de dados textuais. Entretanto, tais operadores falham ao não prover muitas propriedades essenciais requeridas por sistemas de informações modernos, tais como suporte linguístico, ranking dos resultados e indexação [1].

Na Universidade de Moscou foi criado o projeto Tsearch2 [2], no qual os russos Oleg Bartunov e Teodor Sigaev desenvolveram um motor do tipo full text inteiramente integrado ao SGBD PostgreSQL. Além da referida solução de busca textual, foram criados dois poderosos mecanismos de indexação: GiST e GIN.

O Tsearch2 residiu no diretório contrib do PostgreSQL até a sua versão 8.1, sendo finalmente incorporado ao core do SGBD na versão 8.3, tornando-se a opção default para FTS nele.

Para ilustrar, tomemos como exemplo o clássico cadastro de municípios brasileiros. A missão é efetuar buscas textuais (neste caso pelo nome dos municípios) usando FTS.

Eis uma tabela no PostgreSQL populada com as cidades:


        nome         | uf 
---------------------+----
 Abadia de Goiás     | GO
 Abadia dos Dourados | MG
 Abadiânia           | GO
 Abaeté              | MG
 Abaetetuba          | PA
 Abaiara             | CE
 Abaíra              | BA
 Abaré               | BA
 Abatiá              | PR
 Abdon Batista       | SC
 ...

A ideia é fazer com que buscas aproximadas possam ser efetuadas pelo usuário, ou seja, algo mais "poderoso" que um falho operador LIKE.

A primeira modificação será criar uma coluna adicional, chamada de "busca", a fim de armazenar os vetores já normalizados dos termos para ser usada posteriormente nas consultas.


ALTER TABLE municipios ADD busca tsvector;

Eis a estrutura da tabela após a execução do comando:


         Table "public.municipios"
 Column |         Type          | Modifiers 
--------+-----------------------+-----------
 nome   | character varying(50) | 
 uf     | character(2)          | 
 busca  | tsvector              |

Como o foco deste estudo é usar o idioma português, e considerando que o servidor PostgreSQL esteja configurado com a codificação UTF-8 (i.e., Unicode), precisaremos criar algumas funções adicionais. Baseando-se em [3] e [4], criei essas funções:


CREATE FUNCTION to_ascii(bytea, name) RETURNS text AS
 'to_ascii_encname'
 LANGUAGE internal STRICT;

CREATE FUNCTION simples(texto varchar) RETURNS varchar AS
 'select lower(to_ascii(convert_to($1, ''latin1''), ''latin1''))'
 LANGUAGE sql IMMUTABLE STRICT;

Veja o que essa nova função simples() é capaz de fazer:


brasil=# select simples('Pinhão com Açaí');
    simples     
----------------
 pinhao com acai
(1 row)

Ou seja, trata-se de uma função de transformação que normaliza o texto recebido como argumento. Ela será essencial para os passos a seguir.

Podemos usar a função simples() em conjunto com to_tsvector() a fim de produzir o vetor de lexemas (termos processados a serem usados nas buscas). Veja:


brasil=# select to_tsvector(simples('Pinhão com Açaí'));
    to_tsvector    
-------------------
 'aca':3 'pinha':1
(1 row)

Voltando à tabela de municípios, execute o comando SQL a seguir para popular a recém-criada coluna "busca" com o vetor transformado em cada uma de suas tuplas.


UPDATE municipios SET busca = to_tsvector(simples(nome));

Pronto! Agora que os textos foram pré-processados, basta executar as consultas SQL usando o operador especial @@ no PostgreSQL. Veja um exemplo de uso no qual o usuário digitou o texto "sao mateus" na aplicação. A consulta SQL fica assim:


brasil=# select nome, uf from municipios
brasil=# where busca @@ plainto_tsquery(simples('sao mateus'));
          nome          | uf 
------------------------+----
 São Mateus             | ES
 São Mateus do Maranhão | MA
 São Mateus do Sul      | PR
(3 rows)

Note que além do vetor (coluna "busca" do tipo tsvector) é preciso especificar um tipo tsquery, fornecido pelas funções to_tsquery() e plainto_tsquery(). A função simples() foi usada novamente para realizar a transformação do texto informado.

Ótimo! É só isso?

Não... Funcionou, mas o desempenho não pode ser garantido. Na realidade a busca com essa abordagem será mais feliz que se usássemos um ardiloso LIKE "%sao mateus%". (Jogue a primeira pedra quem nunca encontrou um monstrinho desses em uma aplicação!).

Vejamos como o SGBD está tratando a consulta SQL acima:


brasil=# explain select nome, uf from municipios
brasil=# where busca @@ plainto_tsquery(simples('sao mateus'));
                         QUERY PLAN                          
-------------------------------------------------------------
 Seq Scan on municipios  (cost=0.00..163.60 rows=6 width=16)
   Filter: (busca @@ plainto_tsquery('sao mateus'::text))
(2 rows)

Ops, uma varredura sequencial (também conhecido por full scan ou sequential scan) está sendo efetuada! Se você não sabe o que é isso, consulte o seu DBA. Se ele for teu amigo, não irá te xingar.

Bom, e como podemos corrigir isso? Criando um índice! :D

Neste caso específico do FTS no PostgreSQL, temos a escolha de usar uma das duas opções de índices, GiST ou GIN (lembra dos russos?). Cada uma tem a sua aplicação. Escolhi o GIN por esta apresentar maior precisão para esta aplicação. Veja como criar o índice:


CREATE INDEX municipios_gidx ON municipios USING gin(busca);

E agora, será que funcionou? Roda o EXPLAIN novamente! Veja:


brasil=# explain select nome, uf from municipios
brasil=# where busca @@ plainto_tsquery(simples('sao mateus'));
                                  QUERY PLAN                                  
------------------------------------------------------------------------------
 Bitmap Heap Scan on municipios  (cost=4.30..23.49 rows=6 width=16)
   Recheck Cond: (busca @@ plainto_tsquery('sao mateus'::text))
   ->  Bitmap Index Scan on municipios_gidx  (cost=0.00..4.30 rows=6 width=0)
         Index Cond: (busca @@ plainto_tsquery('sao mateus'::text))
(4 rows)

Sucesso! Agora a busca textual com FTS está devidamente indexada no PostgreSQL.

Referências

[1] PostgreSQL 8.3 Documentation - Full Text Search

[2] Tsearch2 - full text search extension for PostgreSQL

[3] PostgreSQL: TO_ASCII & UTF8

[4] PostgreSQL 8.3 Documentation - String Functions and Operators

Buscar palavras do dicionário para registrar domínio

2011-06-15T11:18:00.000-07:00

No início do mês de junho foram finalmente oferecidos registros para o domínio colombiano .CO. Em pouco tempo já haviam um milhão de pedidos de registro! Empresas de destaque na Internet, como Amazon e Twitter foram atraídas pela novidade, e com isso impulsionaram uma corrida desenfreada junto às empresas de registro.

Pensando nas possibilidades de negócio de se garantir um nome de Internet rentável com o sufixo .co, decidi cruzar os dados de um dicionário e fui buscar ajuda ao amigo Shell... Irei mostrar nas linhas a seguir o que consegui com essa brincadeira!

Para iniciar, instalei um dicionário de correção ortográfica do OpenOffice.org, o myspell. No Debian, basta executar o comando a seguir como super usuário:


apt-get install myspell-pt-br

Entre diversos arquivos, o que me interessou foi o pt_BR.dic. Com cerca de 4,5 MB, ele possui mais de 300 mil verbetes da língua portuguesa! Para não correr o risco de danificá-lo, copiei para minha área (i.e., home):


cp /usr/share/hunspell/pt_BR.dic ~

O arquivo é originariamente codificado com ISO-8859-1. Pra me livrar dessa orelhada, o converti para o formato Unicode (i.e., UTF-8) usando o iconv:


iconv -f iso-8859-1 -t utf-8 pt_BR.dic > pt_BR-utf8.dic

Eis um trecho do arquivo pt_BR-utf8.dic:


abacalhoar/akYMjL
abacamartado/D
aba?anar/akYMjL
abacanto/D
abacatada/B
abacataia/B
abacatal/BR
abacate/BP
abacate-do-mato
abacateiral/BR

Não sei precisar que tipo de informação é carregada após o sinal de barra em cada linha do arquivo, mas com certeza não me interessa nesse momento. Vamos removê-la e filtrar apenas as palavras que terminam em "co" com a ajuda de um conhecido amigo, o SED!


sed -e 's/\/.\+$//' -e '/co$/!d' pt_BR-utf8.dic > palavras-co.txt

E eis o arquivo resultante palavras-co.txt, bem mais comportado:


abrodílico
abrotanínico
abrotonínico
absalônico
abscísico
abscíssico
absenteístico
absentístico
absintático
absintêmico

É certo que existem palavras muito esdrúxulas da amada língua portuguesa que jamais seriam usadas num nome da Internet. Usando um outro carinha legal, o AWK, além do comando sort, criei um outro arquivo contendo a quantidade de letras e a palavra, ordenando pela primeira coluna. Eis a mágica:


awk '{print length($0)"\t"$0}' palavras-co.txt | sort -n > palavras-co2.txt

E olha só como ficou o arquivo palavras-co2.txt:


5 dorco
5 écico
5 édico
5 efuco
5 élico
5 êmico
5 emoco
5 ênico
5 épico
5 Érico

Agora resta a parte mais divertida do trabalho: varrer a lista de palavras do dicionário e analisar se cada uma delas já não foi registrada nesse novo domínio ".co". O resultado será um outro arquivo de texto, semelhante ao anterior, mas com a adição de uma terceira coluna, que dirá a disponibilidade do domínio "palavra.co".

Para essa tarefa, usei comandos como while, read, echo e cut do Shell e carinhas legais como o sed (de novo!) e curl. Como consegui a URL de teste? Tive que fuçar nos fontes da página do cointernet.co...

Bom, eis o código resultante em Shell. Ficaria mais bonito eu ter colocado num arquivo Shell Script, mas fiquei com preguiça...


> palavras-co3.txt
while read linha
do
    letras=`echo $linha | cut -f1 -d' '`
    palavra=`echo $linha | cut -f2 -d' '`
    token=`echo $palavra | sed 's/co$/.co/'`
    disp=`curl -s "http://www.opportunity.co/register/whois-lookup.php?oppurl=$token"`
    echo "$palavra"
    echo -e "$linha\t$disp" >> palavras-co3.txt
done < palavras-co2.txt

Olha só que legal o resultado disso: o arquivo de texto resultante (um outro!) trará TRUE caso a palavra esteja disponível para o registro ou FALSE se alguém já tiver tido essa ideia!

Eis um trecho do arquivo palavras-co3.txt:


5 dorco FALSE
5 écico TRUE
5 édico TRUE
5 efuco FALSE
5 élico TRUE
5 êmico TRUE
5 emoco FALSE
5 ênico TRUE
5 épico TRUE
5 Érico TRUE

Legal, funcionou, mas e agora?!

Chuá...! [balde-de-água-na-cabeça]

Ao rodar o script e gerar o arquivo, disparei um tail -f e cheguei à conclusão de que alguém já fez esse trabalho e tratou de registrar tudo quanto é palavra do dicionário de português... (Malditos!)

Aliás, só apareceram como disponíveis as palavras acentuadas, que provavelmente não poderão ser registradas caso haja a versão normalizada delas (ex: "mímico" vs "mimico").

Ou seja, se você tinha esperanças de conseguir algo como bote.co, magi.co, sova.co ou peni.co, perca-as... Ou senão entre em contato com o feladamãe que registrou antes de você e negocie o nome de domínio...

Bom, pelo menos serviu para brincar um pouco com o Shell. :D

Desafio: tente normalizar as palavras do arquivo palavras-co.txt (i.e., deixando tudo em minúsculo e retirando os acentos e caracteres especiais) antes de fazer as consultas WHOIS com o script.

Persistência nas Nuvens com NoSQL

2011-06-06T10:10:00.000-07:00

Um termo recorrente quando se fala em computação em nuvem é a persistência dos dados em bancos do tipo NoSQL, ou seja, em uma forma não-relacional. Essa tecnologia não substitui os consolidados Sistemas Gerenciadores de Bancos de Dados Relacionais (SGBDRs), mas ao invés disso torna-se uma nova ferramenta disponível ao desenvolvedor.

Implantar uma aplicação na nuvem não significa que NoSQL será utilizada. Entretanto, as funcionalidades que essa tecnologia provê são altamente convergentes com as propostas da computação em nuvem: performance, escalabilidade horizontal, alta disponibilidade e flexibilidade.

View more presentations from hjort.

Nesta apresentação são introduzidos conceitos como Computação em Nuvem, Persistência de Dados, Bancos de Dados Relacionais, o movimento NoSQL, o modelo de dados do Bigtable da Google, a arquitetura do Dynamo da Amazon e detalhes técnicos do Apache Cassandra.

Cassandra: Rompendo o Paradigma Relacional com NoSQL

2011-05-09T06:17:00.000-07:00

Apresentação sobre o Apache Cassandra a ser realizada na oficina do Demoiselle Cassandra no CONSEGI 2011.

Buscar diferenças entre diretórios

2010-12-21T06:06:00.000-08:00

Eis um problema que já tive que enfrentar diversas vezes: alterar um parâmetro de configuração que não sabia onde estava! Ou ainda: investigar o que foi alterado em algo que antes funcionava e passou a dar problema.

Como na informática tudo gira em torno de 0s e 1s, se algo mudou, tem que haver diferença entre arquivos e ou diretórios. Digamos que você possua dois diretórios contendo parâmetros de configuração de determinado software. Um deles funciona, o outro não. Como saber o que há de diferente entre eles?

Richard Stallman, junto com outros colaboradores da GNU nos proveu o utilitário diff no Linux (ou melhor, GNU/Linux), o qual permite a comparação entre dois arquivos de texto ou binários. O problema é que esse comando precisa ser invocado para cada par de arquivos, o que pode se tornar um complexo "trabalho de estagiário" dependendo da quantidade de subdiretórios e arquivos considerados.

Para atacar essa questão, criei o script em Shell diffdirs.sh contendo o código a seguir:


#!/bin/bash

if [ $# -ne 2 ]
then
    echo "Uso: $0 <dir1> <dir2>"
    exit 1
fi

dir1="$1"
dir2="$2"
list="/tmp/$$.list"
cont="/tmp/$$.cont"

cd $dir1
find -type f | sed 's/^\.\///' > $list
cd - > /dev/null

echo "Analisando diferenças entre diretórios:"
echo "- $dir1"
echo "- $dir2"

while read arq
do
    if ! diff $dir1/$arq $dir2/$arq > $cont
    then
        echo -e "\n$arq:"
        cat $cont
    fi
done < $list

rm -f $list $cont

exit 0

No caso em questão, precisava saber em qual diretório e arquivo residia uma configuração específica da IDE Eclipse. Como não dispunha de ambiente gráfico, a opção foi fazer tudo em modo texto. E esse script resolveu o problema! Veja o resultado exibido após a sua execução:


$ ./diffdirs.sh 
Uso: ./diffdirs.sh <dir1> <dir2>

$ ./diffdirs.sh ws35a/ ws35b/


Analisando diferenças entre diretórios:
- ws35a
- ws35b

.metadata/.plugins/org.eclipse.jdt.core/nonChainingJarsCache:
Os arquivos binários ws35a/.metadata/.plugins/org.eclipse.jdt.core/nonChainingJarsCache e ws35b/.metadata/.plugins/org.eclipse.jdt.core/nonChainingJarsCache são diferentes

.metadata/.plugins/org.eclipse.core.runtime/.settings/org.maven.ide.eclipse.prefs:
1c1
< #Tue Dec 21 09:47:38 AMT 2010
---
> #Tue Dec 21 09:48:27 AMT 2010
3d2
< eclipse.m2.downloadSources=true
10d8
< eclipse.m2.downloadJavadoc=true

Agora é só copiar o script para um diretório no PATH do usuário. :D

Assembling a local Cassandra cluster in Linux

2010-09-24T07:17:00.000-07:00

In order to build a Apache Cassandra [1] cluster exclusively for availability and replication testings, here is a simple solution, based on a single Linux instance, with no virtualization at all.

The idea is to initialize every node, run a testing client, and then manually kill some nodes processes in order to check the service continuous availability and data replication with several replication factors.

Cassandra's last stable version at this post date was 0.6.5, and this is the released considered in the study.

Thinking of a cluster using 3 nodes, we'll create the following file system structure on Linux:


/opt/apache-cassandra-0.6.5/nodes
|
|-- node1
|   |-- bin
|   |-- conf
|   |-- data
|   |-- log
|   `-- txs
|
|-- node2
|   |-- bin
|   |-- conf
|   |-- data
|   |-- log
|   `-- txs
|
`-- node3
    |-- bin
    |-- conf
    |-- data
    |-- log
    `-- txs

That is, each node has its own set of directories, namely:
- bin: binaries (mostly Shell Scripts) tuned for the instance
- conf: configuration files for the node
- data: database binary files
- log: system log in text format
- txs: transaction logs (i.e. commit logs)

In the following sections the steps will be described in detail.

Cassandra installation

First of all, get the required packages for Cassandra from its download site. Extract the file contents into /opt/apache-cassandra-0.6.5/ directory (you'll need administrator privileges in order to do so). Make sure your regular user owns this entire directory tree (use chown if needed).

Certify that you already has Java Virtual Machine (JVM) installed on the system, at least version 1.6. Cassandra is implemented in Java, so it needs JVM to be executed.

Supplying additional network interfaces

As we are building a pseudo-cluster, consider there are no more network interfaces than existing ones. Since Cassandra is a distributed system, their participant nodes must have an exclusive IP address each one. Fortunately we can simulate it on Linux using aliases to the existent loopback interface (i.e., "lo").

Therefore, using root user, we'll issue the instructions below in order to create the aliases lo:2 and lo:3, respectively pointing to 127.0.0.2 and 127.0.0.3:


# ifconfig lo:2 127.0.0.2 up
# ifconfig lo:3 127.0.0.3 up

You can check the overcome of these instructions by invoking ifconfig using no arguments:


$ ifconfig
lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:30848 errors:0 dropped:0 overruns:0 frame:0
          TX packets:30848 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:2946793 (2.9 MB)  TX bytes:2946793 (2.9 MB)

lo:2      Link encap:Local Loopback
          inet addr:127.0.0.2  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1

lo:3      Link encap:Local Loopback
          inet addr:127.0.0.3  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1

That way, node1's IP is 127.0.0.1, node2's 127.0.0.2, and so forth.

Registering node hostnames locally

Instead of using IP addresses in the sequential steps, we'll assign hostnames to them and use those names when referencing a cluster node.

As we have no DNS in this configuration, we need to edit /etc/hosts file and assign those IP mappings there. Change this file according to the example below:


/etc/hosts:
127.0.0.1    localhost    node1
127.0.0.2    node2
127.0.0.3    node3

From now on, we'll call the nodes simply as node1, node2, and node3. In order to finally test hostnames, try pinging the nodes as exemplified:


$ ping node2
PING node2 (127.0.0.2) 56(84) bytes of data.
64 bytes from node2 (127.0.0.2): icmp_seq=1 ttl=64 time=0.018 ms
64 bytes from node2 (127.0.0.2): icmp_seq=2 ttl=64 time=0.015 ms
^C
--- node2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.015/0.016/0.018/0.004 ms

Creating the first node structure

We'll first create the directory structure for node1 and after replicate it to remaining nodes. Switch to Cassandra's root directory, create subdirectories and copy files by issuing the following shell commands using your regular Linux user:


$ cd /opt/apache-cassandra-0.6.5/
$ mkdir -p nodes/node1
$ cp -R bin/ conf/ nodes/node1/
$ cd nodes/node1

The next step is to modify some default settings in the configuration files. Here I opted to use relative paths instead of absolute paths for simplification purposes.

Edit a single line in conf/log4j.properties file.


# Edit the next line to point to your logs directory
log4j.appender.R.File=./log/system.log

Now in conf/storage-conf.xml several tag values must be changed. Be sure to modify the correct tags, as indicated below:


  <Keyspaces>
    <Keyspace Name="Keyspace1">
      ...

      <ReplicationFactor>2</ReplicationFactor>
      ...

    </Keyspace>
  <Keyspaces>
  ...

  <CommitLogDirectory>./txs</CommitLogDirectory>
  <DataFileDirectories>
      <DataFileDirectory>./data</DataFileDirectory>
  </DataFileDirectories>
  ...

  <ListenAddress>node1</ListenAddress>
  <StoragePort>7000</StoragePort>
  ...

  <ThriftAddress></ThriftAddress>
  <ThriftPort>9160</ThriftPort>
  <ThriftFramedTransport>false</ThriftFramedTransport>
  ...

As we are building a special directory arrangement, we'll need to adequate some paths in the scripts. Thus, edit the file bin/cassandra.in.sh by providing the lines below appropriately:


for jar in $cassandra_home/../../lib/*.jar; do
    CLASSPATH=$CLASSPATH:$jar
done

The first node is finally concluded. So, let's turn our attentions to the other two nodes.

Creating the remaining nodes

We are gonna take advantage of the first node directory structure to create the two remaining nodes. Thus, issue the instructions below in order to perform the cloning:


$ cd /opt/apache-cassandra-0.6.5/nodes/
$ mkdir node2 node3
$ cp -R node1/* node2
$ cp -R node1/* node3

When you're done, call a tree command (if available) as illustrated below to check the whole structure:


$ tree -L 2
.
|-- node1
|   |-- bin
|   `-- conf
|-- node2
|   |-- bin
|   `-- conf
`-- node3
    |-- bin
    `-- conf

If you compare this structure to the other one in the beginning of this article, you should note that there are some subdirectories missing (i.e., log, data, txs). That's because they will be created automatically when the server starts up the first time.

Final settings on nodes

The first node is deemed a contact point, i.e., a seed. Other nodes should start the Gossip-based protocol by connecting to it.

Besides, we should make specific settings on each node. Firstly, each node must listen to it's own hostname (i.e., node1, node2, node3). Thus, edit the nodeX/conf/storage-conf.xml for each node by replacing the line below (this example applies to node2):


  <ListenAddress>node2</ListenAddress>

Monitoring on Cassandra is provided through JMX, listening to every host by default on TCP port 8080. As we are building the cluster on the same real instance, this is not possible anymore. Thus, JMX interfaces must be bound to the same host, but we must change the port on each node. Therefore, first node will be listening on 8081, node2 on 8082, and node3 on 8083.

This parameter is configured in nodeX/bin/cassandra.in.sh, so we must change only the port as exemplified below (applying to node2):


# Arguments to pass to the JVM
JVM_OPTS=" \
        -ea \
        -Xms1G \
        -Xmx1G \
        -XX:+UseParNewGC \
        -XX:+UseConcMarkSweepGC \
        -XX:+CMSParallelRemarkEnabled \
        -XX:SurvivorRatio=8 \
        -XX:MaxTenuringThreshold=1 \
        -XX:+HeapDumpOnOutOfMemoryError \
        -Dcom.sun.management.jmxremote.port=8082 \
        -Dcom.sun.management.jmxremote.ssl=false \
        -Dcom.sun.management.jmxremote.authenticate=false"

Starting up every server

For this step it is very interesting to open a new Linux terminal for each node. Thus, issue the instructions below:


$ node1/bin/cassandra -f

$ node2/bin/cassandra -f

$ node3/bin/cassandra -f

Pay attention to the console output on each node. A simple misconfiguration will be related there. Also note the relationship between the nodes. They must discover one another automatically in seconds.

Check services availability

Since every node in the cluster is up and running, their respective TCP ports must appear to the system when executing a netstat command.

In order to check listening TCP ports on Cassandra, one must search for 9160 (Thrift service), 7000 (internal storage), and 808X (JMX interface). Take a look at the example below of a successful cluster startup.


$ netstat -lptn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN      -             
tcp6       0      0 127.0.0.3:9160          :::*                    LISTEN      8520/java     
tcp6       0      0 127.0.0.2:9160          :::*                    LISTEN      8424/java     
tcp6       0      0 127.0.0.1:9160          :::*                    LISTEN      8336/java     
tcp6       0      0 :::46954                :::*                    LISTEN      8424/java     
tcp6       0      0 :::53418                :::*                    LISTEN      8336/java     
tcp6       0      0 :::49035                :::*                    LISTEN      8520/java     
tcp6       0      0 :::80                   :::*                    LISTEN      -             
tcp6       0      0 :::42737                :::*                    LISTEN      8520/java     
tcp6       0      0 :::8081                 :::*                    LISTEN      8336/java     
tcp6       0      0 :::8082                 :::*                    LISTEN      8424/java     
tcp6       0      0 :::8083                 :::*                    LISTEN      8520/java     
tcp6       0      0 :::60310                :::*                    LISTEN      8424/java     
tcp6       0      0 :::46167                :::*                    LISTEN      8336/java     
tcp6       0      0 ::1:631                 :::*                    LISTEN      -             
tcp6       0      0 127.0.0.3:7000          :::*                    LISTEN      8520/java     
tcp6       0      0 127.0.0.2:7000          :::*                    LISTEN      8424/java     
tcp6       0      0 127.0.0.1:7000          :::*                    LISTEN      8336/java

The last test was from a network perspective. You might want to ask Cassandra whether these nodes are talking to each other and participating on a healthy partitioning scheme. In order to do so, another checking is to invoke nodetool's ring command.


$ cd /opt/apache-cassandra-0.6.5/

$ ./bin/nodetool -h localhost -p 8081 ring
Address       Status     Load          Range                                      Ring
                                       142865723918937898194528652808268231850  
127.0.0.1     Up         3,1 KB        39461784941927371686416024510057184051     |<--|
127.0.0.3     Up         3,1 KB        54264004217607518447601711663387808864     |   |
127.0.0.2     Up         2,68 KB       142865723918937898194528652808268231850    |-->|

Here it is, the local cluster is up and running with three nodes! :D

You can also check Cassandra's availability by using its simple client application:


$ ./bin/cassandra-cli --host node1 --port 9160
Connected to: "Test Cluster" on node1/9160
Welcome to cassandra CLI.

cassandra> set Keyspace1.Standard1['rowkey']['column'] = 'value'
Value inserted.

cassandra> get Keyspace1.Standard1['rowkey']['column']          
=> (column=636f6c756d6e, value=value, timestamp=1285273581745000)

cassandra> get Keyspace1.Standard1['rowkey']          
=> (column=636f6c756d6e, value=value, timestamp=1285273581745000)
Returned 1 results.

cassandra> del Keyspace1.Standard1['rowkey']                    
row removed.

As we set replication factor to 2 for this given keyspace Keyspace1, it is expected that each value saved in the Cassandra cluster is replicated to two nodes.

Do you really believe it is happening? If not, save another value (using set) and then kill one of the three node processes. Do not kill more than one! Then, retrieve that key again (using get). The system should be capable of automatically recovering from this intentional disaster.

You can easily make a node participate in the cluster again just by executing its respective bin/cassandra command. Take a look at all console outputs after every step, they are very interesting!

References

[1] Apache Cassandra

PGDay Manaus 2010

2010-09-17T10:30:00.000-07:00

No dia 15 de outubro ocorrerá o PGDay Manaus 2010 na Universidade Federal do Amazonas (UFAM), em Manaus.

Trata-se de um dia inteiro dedicado ao SGBD PostgreSQL, contendo oficinas, palestras e mostras de trabalhos.

Para maiores informações, consulte o site do evento:
http://www.postgresql.org.br/eventos/pgday/am/2010

As inscrições já estão abertas nessa mesma página, participe! :D

Monitorando serviços com SNMP e Zabbix

2010-09-14T08:59:00.000-07:00

Inicialização do servidor SNMP e coleta da rede

Entre no terminal da máquina virtual e inicialize o serviço SNMP nela através do comando:

# service snmpd restart

Ainda no terminal, execute os comandos snmpget, snmpgetnext e snmpwalk a fim de testar o cliente e servidor SNMP fornecido pela implementação do Net-SNMP:

# snmpget -v 1 -c public localhost sysDescr.0

# snmpget -v 1 -c public localhost system.sysName.0

# snmpget -v 1 -c public localhost ifDescr.2

# snmpgetnext -v 1 -c public localhost interfaces

# snmpgetnext -v 1 -c public localhost host

# snmpgetnext -v 1 -c public localhost snmpV2

# snmpwalk -v 1 -c public localhost system

# snmpwalk -v 1 -c public localhost interfaces

Repita os comandos anteriores utilizando os parâmetros -On, -Os e -Of.

Volte ao frontend do Zabbix e inclua um item com as seguintes configurações:

Description: eth0 in/s
Type: SNMPv1 agent
SNMP OID: interfaces.ifTable.ifEntry.ifInOctets.2
SNMP community: public
SNMP port: 161
Key: eth0.in
Type of information: Numeric (unsigned)
Store value: Delta (speed per second)

Repita a operação para incluir este outro indicador:

Description: eth0 out/s
Type: SNMPv1 agent
SNMP OID: ifOutOctets.2
SNMP community: public
SNMP port: 161
Key: eth0.out
Type of information: Numeric (unsigned)
Store value: Delta (speed per second)

Aguarde alguns minutos e então acesse o menu Monitoring > Latest data e abra os gráficos correspondente as coletas dos indicadores recém-criados:

Coleta da carga da CPU e gráficos customizados

Entre no terminal da máquina virtual e explore a UCD-MIB executando os comandos a seguir:

# snmpwalk -v 1 -c public localhost ucdavis | less

# snmpbulkget -v 2c -c public localhost -C r3 laNames laLoadInt -Os

Observe a carga média dos processadores executando as instruções a seguir:

# uptime

# snmpwalk -v 1 -c public localhost laLoadInt -Os

Provoque o aumento no consumo de CPU executando diversas vezes o comando abaixo:

# cat /dev/urandom | md5sum &

Dica: para cancelar a execução desses processos, utilize os comandos jobs e kill do Linux.

De volta ao frontend do Zabbix, cadastre o seguinte indicador para o host:

Description: cpu load-1
Type: SNMPv1 agent
SNMP OID: laLoadInt.1 (pode ser OID completo – textual ou numérico)
SNMP community: public
SNMP port: 161
Key: laLoadInt.1
Type of information: Numeric (float)
Use multiplier: Custom multiplier
Custom multiplier: 0.01

Utilizando a funcionalidade de clonagem de indicadores, crie mais dois indicadores alterando apenas os campos indicados abaixo:

Description: cpu load-5
SNMP OID: laLoadInt.2
Key: laLoadInt.2

Description: cpu load-15
SNMP OID: laLoadInt.3
Key: laLoadInt.3

Acesse o menu Configuration > Hosts e crie um novo gráfico contendo os três indicadores de carga de CPU recém incluídos para o host:

Acesse o menu Monitoring > Latest data e o menu Monitoring > Graphs para acompanhar a evolução da coleta destes indicadores:

Referências:

VirtualBox: http://www.virtualbox.org/
Net-SNMP: http://www.net-snmp.org/
Zabbix: http://www.zabbix.org/
Ubuntu VirtualBox: https://help.ubuntu.com/community/VirtualBox
Zabbix Appliance: http://www.zabbix.com/documentation/1.8/manual/installation/appliance

Testando Zabbix Appliance com VirtualBox

2010-09-14T08:14:00.000-07:00

Baixe a partir do endereço a seguir o pacote Debian adequado a sua distribuição Linux:

http://www.virtualbox.org/wiki/Linux_Downloads

Instale o pacote via DPKG utilizando as instruções abaixo:

# dpkg -i virtualbox-*.deb

# apt-get -f install

Baixe a imagem ISO do Zabbix Appliance a partir do endereço a seguir:

http://www.zabbix.com/download.php

Execute o VirtualBox e crie uma nova máquina utilizando os parâmetros seguintes:

Nome: Zabbix 1.8
Sistema Operacional: Linux
Versão: openSUSE
Memória: 512 MB
Disco rígido: (nenhum)

Clique na máquina recém-criada e associe a imagem ISO do Zabbix Appliance no drive de CD:

Configure a rede para utilizar a opção “host-only” e a interface “vboxnet0”:

Inicie a nova máquina virtual e aguarde o completo carregamento do sistema operacional.

No prompt que se abrirá, entre com o usuário “root” e a senha “zabbix”.

Carregue o mapa de teclados adequado e confira o endereço IP associado à máquina virtual através dos comandos a seguir:

# loadkeys br-abnt2

# ifconfig eth0

Anote o endereço IP exibido (ex: 192.168.56.101).

Abra no navegador o endereço HTTP correspondente ao frontend do Zabbix. Ex:

http://192.168.56.101/zabbix/

Entre na aplicação utilizando o login “Admin” e a password “zabbix”:

Acesse o menu Configuration > Hosts e crie um novo host utilizando essas configurações:

Name: Local
Groups: Linux servers
IP address: 127.0.0.1
Connect to: IP address

Acesse os indicadores do host e inclua um novo conforme indicado a seguir:

Description: cpu load
Type: Zabbix agent
Key: system.cpu.load
Type of information: Numeric (float)

Aguarde alguns minutos e então acesse o menu Monitoring > Latest data, procurando pelos dados históricos e gráficos gerados pela coleta do indicador “cpu load”.

Referências:

VirtualBox: http://www.virtualbox.org/
Zabbix: http://www.zabbix.org/
Ubuntu VirtualBox: https://help.ubuntu.com/community/VirtualBox
Zabbix Appliance: http://www.zabbix.com/documentation/1.8/manual/installation/appliance

Simplifying Cassandra interactions with Helena

2010-09-02T13:35:00.000-07:00

Introduction

In the previous article "Talking to Cassandra using Java and DAO pattern" [1] we saw an introduction to NoSQL-based databases and especially one called Apache Cassandra [2]. After a quick explanation of its data model, we analyzed what it takes to implement a Java code which goal was to access and yet modify key-value pairs stored on a Cassandra keyspace. At that time we used a low-level interface API named Thrift.

You might have noticed that a large amount of code was needed to build an entire and yet simple persistence class, namely a structure based on Data Access Object design pattern. If we compare it to currently most used persistence technologies and frameworks in Java (i.e. JPA and Hibernate), we would probably get crazy!

Is this the price we pay for high-availability and scalability when changing from a traditional relational database model to a newly key-value data store? Poor developers... Well, that's not the end, fortunately! There are already available several high level clients for Cassandra in multiple programming languages (see [3]).

One of these clients, HelenaORM [4] (created by Marcus Thiesen), is the study subject this time. As this library is addressed to Java language, our aim will be to implement the persistence of a simple object using Helena's support.

Creating the entity

As in the other article, we'll choose the "group" entity to exemplify our codes. This time, in addition to the plain old Java class elements, in the Group class we need to annotate it with @HelenaBean and @KeyProperty.

The annotation @HelenaBean is used to specify keyspace and column family for the class, whereas @KeyProperty must indicate which field in the class should be considered the row key in Cassandra - currently annotation only works at getter or setter methods, not in the variable itself. Both annotations are analogous to JPA's @Entity and @Id annotations.

Take a look at the Group entity class properly prepared to be used by HelenaORM:


@HelenaBean(keyspace="ContactList", columnFamily="Groups")
public class Group {

  private Integer id;
  private String name;

  public Group() {
  }

  public Group(Integer id, String name) {
      this.id = id;
      this.name = name;
  }

  @KeyProperty
  public Integer getId() {
      return id;
  }
  public String getName() {
      return name;
  }
  public void setId(Integer id) {
      this.id = id;
  }
  public void setName(String name) {
      this.name = name;
  }

  @Override
  public String toString() {
      return "Group [id=" + id + ", name=" + name + "]";
  }

}

Creating the tests

The most interesting new is about to come: with Helena there's no need of traditional building a DAO interface and class!

Helena brings a factory HelenaORMDAOFactory designed to create ready-to-use DAO classes type-safely pointed to a given entity. The classes it produces, of HelenaDAO type, provides these methods: insert(), delete(), and get(). They are out-of-box implementations respectively for inserting (or editing), removing, and retrieving entire object instances from Cassandra.

So, here is the corresponding unitary test class in JUnit reserved for invoking inserts, deletions, and retrievals of Java object instances into Cassandra with the support of Helena:


public class GroupTest {

    static HelenaORMDAOFactory factory;
    private HelenaDAO<Group> dao;

    private static final Integer GROUP_ID = 123;
    private static final String GROUP_NAME = "Test Group";

    @BeforeClass
    public static void setUpBeforeClass() throws Exception {
        factory = HelenaORMDAOFactory.withConfig(
                "localhost", 9160, SerializeUnknownClasses.YES);
    }

    @AfterClass
    public static void tearDownAfterClass() throws Exception {
        factory = null;
    }

    @Before
    public void setUp() throws Exception {
        dao = factory.makeDaoForClass(Group.class);
    }

    @After
    public void tearDown() throws Exception {
        dao = null;
    }
    
    @Test
    public void testSave() {

        System.out.println("GroupDAOTest.testSave()");

        Group group = new Group();
        group.setId(GROUP_ID);
        group.setName(GROUP_NAME);

        System.out.println("Saving group: " + group);
        dao.insert(group);
        Assert.assertTrue(true);

        Group retrieved = dao.get(GROUP_ID.toString());
        System.out.println("Retrieved group: " + retrieved);
        Assert.assertNotNull(retrieved);
        Assert.assertEquals(GROUP_ID, retrieved.getId());
        Assert.assertEquals(GROUP_NAME, retrieved.getName());
    }

    @Test
    public void testRetrieve() {

        System.out.println("GroupDAOTest.testRetrieve()");

        System.out.println("Saving groups");
        for (int i = 1; i <= 10; i++) {
            Group group = new Group();
            group.setId(GROUP_ID * 100 + i);
            group.setName(GROUP_NAME + " " + i);
            dao.insert(group);
        }

        List<Group> list = dao.getRange("", "", 10);
        System.out.println("Retrieving groups");
        Assert.assertNotNull(list);
        Assert.assertFalse(list.isEmpty());
        Assert.assertTrue(list.size() >= 10);

        System.out.println("Retrieved list:");
        for (Group group : list) {
            System.out.println("- " + group);
        }
    }

    @Test
    public void testRemove() {

        System.out.println("GroupDAOTest.testRemove()");

        Group group = new Group();
        group.setId(GROUP_ID);
        group.setName(GROUP_NAME);

        System.out.println("Saving group: " + group);
        dao.insert(group);

        System.out.println("Removing group: " + group);
        dao.delete(group);
        Assert.assertTrue(true);

        Group retrieved = dao.get(GROUP_ID.toString());
        System.out.println("Retrieved group: " + retrieved);
        Assert.assertNull(retrieved);
    }

}

Checking the results

After properly executing the tests, you should check the keyspace contents inside Cassandra. In order to do that, you can use Cassandra's client by issuing the instructions described below:


cassandra> count ContactList.Groups['12305']    
1 column

cassandra> get ContactList.Groups['12305']      
=> (column=name, value=Test Group 5, timestamp=1283287882927)
Returned 1 result.

cassandra> get ContactList.Groups['12305']['name']
=> (column=name, value=Test Group 5, timestamp=1283287882927)

Conclusions

Traditional relational databases took more than 20 years of evolution to reach the state they are now. Aside, object-oriented programming languages and techniques conquered companies and developers forcing the creation of persistence frameworks in order to link both worlds. Soon as Internet usage grew, RDBMSs were not able to efficiently scale, thus a new paradigm was conceived (or reborn): the key-value distributed database model!

Cassandra, one of those distributed databases, is not trivial to talk to, from a developer perspective. In opposition to use a low-level interface API (i.e. Thrift), a lot of high-level clients were created by individual developers (thanks to open source initiatives!), and one of them was HelenaORM.

Thus, in the present article we saw how to leverage and simplify our Java code which handles persistence in Cassandra with the aid of HelenaORM libraries. A lot of work was reduced, isn't it? :D

References

[1] Talking to Cassandra using Java and DAO pattern
[2] Apache Cassandra
[3] Cassandra high level clients
[4] HelenaORM

Talking to Cassandra using Java and DAO pattern

2010-09-01T11:54:00.000-07:00

Introduction

It is definitely unavoidable and not just another hype this wave of NoSQL - an alternative to rigid persistence models provided by RDBMS (Relational Database Management Systems). In a connected world that needs more and more data in a continuous growth and speed, it is proved by huge technology companies like Google that changes need to be done in order to accomodate this. One of these turns was indeed NoSQL databases, such as Bigtable, Hypertable, and Cassandra. The latter will be the subject of this article.

As described in its home page [1]:

"The Apache Cassandra Project develops a highly scalable second-generation distributed database, bringing together Dynamo's fully distributed design and Bigtable's ColumnFamily-based data model. Cassandra was open sourced by Facebook in 2008, and is now developed by Apache committers and contributors from many companies."

So far, Cassandra is in use at Digg, Facebook, Twitter, Reddit, Rackspace, Cloudkick, Cisco, SimpleGeo, Ooyala, OpenX, and more companies that have large, active data sets. The largest production cluster has over 100 TB of data in over 150 machines!

Therefore, let's take a walk-through on how can this new storing and programming model be achieved, particularly using a client on Java language.

Database keyspace setup

We are considering Cassandra's stable version on the time of this document: 0.6.5. Yet, we'll run the server in single mode for simplification purposes. Remember that JDK 1.6 (Java Development Kit) is needed to run Cassandra server.

The first step on this study will be to include the keyspace "ContactList". In order to to this, add the following lines into CASSANDRA_HOME/conf/storage-conf.xml, inside <Keyspaces> tag:


<Keyspace Name="ContactList">
  <ColumnFamily CompareWith="UTF8Type" Name="Groups"/>
  <ColumnFamily CompareWith="UTF8Type" Name="Contacts"/>
  <ReplicaPlacementStrategy>org.apache.cassandra.locator.RackUnawareStrategy</ReplicaPlacementStrategy>
  <ReplicationFactor>1</ReplicationFactor>
  <EndPointSnitch>org.apache.cassandra.locator.EndPointSnitch</EndPointSnitch>
</Keyspace>

Start (or restart) your Cassandra server (by issuing "CASSANDRA_HOME/bin/cassandra -f") after editing that file. The proper data files and index structures should be created.

Please consider reading Cassandra's Getting Started guide [2] for further information on this process.

Once the server is online, check the new keyspace availability by using the client application invoking the following command:

CASSANDRA_HOME/bin/cassandra-cli --host localhost --port 9160

The resulting expected screen is shown below:


Connected to: "Test Cluster" on localhost/9160
Welcome to cassandra CLI.

Type 'help' or '?' for help. Type 'quit' or 'exit' to quit.
cassandra>

You could then try to show available keyspaces on the server, as illustrated:


cassandra> show keyspaces
ContactList
Keyspace1
system

Note that "ContactList" should be included. If not, you should review the previous steps.

Now invoke the instruction below in order to explain detailed information on the new keyspace:


cassandra> describe keyspace ContactList
ContactList.Groups
Column Family Type: Standard
Columns Sorted By: org.apache.cassandra.db.marshal.UTF8Type@33b121

Column Family Type: Standard
Column Sorted By: org.apache.cassandra.db.marshal.UTF8Type
flush period: null minutes
------
ContactList.Contacts
Column Family Type: Standard
Columns Sorted By: org.apache.cassandra.db.marshal.UTF8Type@1b22920

Column Family Type: Standard
Column Sorted By: org.apache.cassandra.db.marshal.UTF8Type
flush period: null minutes
------

OK, apparently our key-value storage is working well. It's time for coding! :D

Starting the use case

We'll start by implementing a simple "group" entity, represented in Java by a POJO-based class named Group with the content below:


public class Group {

  private Integer id;
  private String name;

  public Group() {
  }

  public Group(Integer id, String name) {
      this.id = id;
      this.name = name;
  }

  public Integer getId() {
      return id;
  }
  public String getName() {
      return name;
  }
  public void setId(Integer id) {
      this.id = id;
  }
  public void setName(String name) {
      this.name = name;
  }

  @Override
  public String toString() {
      return "Group [id=" + id + ", name=" + name + "]";
  }

}

Concerns must be separated in different layers. In order to persist the "group" entity, it is highly recommended to have a DAO pattern-based interface. That interface, named IGroupDAO, is destinated to exclusively handle operations on that entity. Here is its corresponding Java code:


public interface IGroupDAO {

  void startup();

  void shutdown();

  List<Group> findAll();

  Group findById(Integer id);

  void save(Group group);

  void remove(Group group);

}

Simply put, save() and remove() methods are enough to perform modifications on the data, whereas findAll() and findById() are reserved for retrieving purposes. You might then ask: "what da hell are startup() and shutdown() doing there"? Hold on, they are a necessary "gambiarra" to show stuff working and might not be inserted on a real application. :)

Developers, developers, developers: do not forget to write tests before actual codes..! Test Driven Development (TDD) is definitely important and worthful besides being so funny to use at coding time!

As we are using Java, let's create unitary a test against that DAO interface using the JUnit [3] testing framework. The resulting class, named GroupDAOTest has the code shown below:


public class GroupDAOTest {

  private static IGroupDAO dao;

  private static final Integer GROUP_ID = 123;
  private static final String GROUP_NAME = "Test Group";

  @BeforeClass
  public static void setUpBeforeClass() throws Exception {
      dao = new GroupDAO();
      dao.startup();
  }

  @AfterClass
  public static void tearDownAfterClass() throws Exception {
      dao.shutdown();
      dao = null;
  }

  @Test
  public void testSave() {

      System.out.println("GroupDAOTest.testSave()");

      Group group = new Group();
      group.setId(GROUP_ID);
      group.setName(GROUP_NAME);

      System.out.println("Saving group: " + group);
      dao.save(group);
      Assert.assertTrue(true);

      Group retrieved = dao.findById(GROUP_ID);
      System.out.println("Retrieved group: " + retrieved);
      Assert.assertNotNull(retrieved);
      Assert.assertEquals(GROUP_ID, retrieved.getId());
      Assert.assertEquals(GROUP_NAME, retrieved.getName());
  }

  @Test
  public void testRetrieve() {

      System.out.println("GroupDAOTest.testRetrieve()");

      System.out.println("Saving groups");
      for (int i = 1; i <= 10; i++) {
          Group group = new Group();
          group.setId(GROUP_ID * 100 + i);
          group.setName(GROUP_NAME + " " + i);
          dao.save(group);
      }

      List<Group> list = dao.findAll();
      System.out.println("Retrieving groups");
      Assert.assertNotNull(list);
      Assert.assertFalse(list.isEmpty());
      Assert.assertTrue(list.size() >= 10);

      System.out.println("Retrieved list:");
      for (Group group : list) {
          System.out.println("- " + group);
      }
  }

  @Test
  public void testRemove() {

      System.out.println("GroupDAOTest.testRemove()");

      Group group = new Group();
      group.setId(GROUP_ID);
      group.setName(GROUP_NAME);

      System.out.println("Saving group: " + group);
      dao.save(group);

      System.out.println("Removing group: " + group);
      dao.remove(group);
      Assert.assertTrue(true);

      Group retrieved = dao.findById(GROUP_ID);
      System.out.println("Retrieved group: " + retrieved);
      Assert.assertNull(retrieved);
  }

}

The goal of this GroupDAOTest test class is to invoke the methods belonging to IGroupDAO interface. For each DAO functionality a method on GroupDAOTest must be created and annotated with @Test. The expected results might be tested using Assert class.

Actual persistence

Base codes were built along with testing classes. However, persistence against Cassandra is not being made yet. Thus, how could we get access to a Cassandra database from a client perspective?

Cassandra uses a more general API called Thrift [4] for its external client-facing interactions. Cassandra's main API/RPC/Thrift port is 9160. Thrift supports a wide variety of languages so you can code your application to use Thrift directly (take a look at [5]), or use a high-level client where available.

Of course we're gonna use the low-level Thrift interface. :)

The first thing to do is to add several Java libraries related to Cassandra and Thrift to the project's classpath. They are released along with Cassandra binaries, especifically at CASSANDRA_HOME/lib directory.

And now we finally implement the GroupDAO class, which must provide actual instructions assigned by IGroupDAO interface.

In this code, we'll need to manually handle the connection to Cassandra through its binary protocol on TCP port 9160. This is exactly where startup() and shutdown() get in.

If you're still not familiar with terms used at key-value databases, such as keystore, column families, etc, don't worry. I suggest you further reading of some articles about Cassandra's Data Model [6, 7]. For the moment, just know that Thrift interface for Java provides support classes like Cassandra.Client, ColumnParent, ColumnOrSuperColumn, KeyRange, SliceRange, and SlicePredicate.

Take a look at the code below for GroupDAO implementation class:


public class GroupDAO implements IGroupDAO {

  private static final String KEYSPACE = "ContactList";
  private static final String COLUMN_FAMILY = "Groups";
  private static final String ENCODING = "utf-8";

  private static TTransport tr = null;

  /**
   * Close the connection to the Cassandra Database.
   */
  private static void closeConnection() {
      try {
          tr.flush();
          tr.close();
      } catch (TTransportException exception) {
          exception.printStackTrace();
      }
  }

  /**
   * Open up a new connection to the Cassandra Database.
   *
   * @return the Cassandra Client
   */
  private static Cassandra.Client setupConnection() {
      try {
          tr = new TSocket("localhost", 9160);
          TProtocol proto = new TBinaryProtocol(tr);
          Cassandra.Client client = new Cassandra.Client(proto);
          tr.open();
          return client;
      } catch (TTransportException exception) {
          exception.printStackTrace();
      }
      return null;
  }

  private Cassandra.Client client;

  @Override
  public void startup() {
      this.client = setupConnection();
  }

  @Override
  public void shutdown() {
      closeConnection();
  }

  @Override
  public List<Group> findAll() {
      List<Group> list = new ArrayList<Group>();
      try {
          KeyRange keyRange = new KeyRange();
          keyRange.setStart_key("");
          keyRange.setEnd_key("");

          SliceRange sliceRange = new SliceRange();
          sliceRange.setStart(new byte[] {});
          sliceRange.setFinish(new byte[] {});

          SlicePredicate slicePredicate = new SlicePredicate();
          slicePredicate.setSlice_range(sliceRange);

          ColumnParent columnParent = new ColumnParent(COLUMN_FAMILY);
          List<KeySlice> keySlices = client.get_range_slices(KEYSPACE,
                  columnParent, slicePredicate, keyRange,
                  ConsistencyLevel.ONE);

          if (keySlices == null || keySlices.isEmpty())
              return list;

          for (KeySlice keySlice : keySlices) {

              Group group = new Group();
              group.setId(Integer.parseInt(keySlice.getKey()));

              for (ColumnOrSuperColumn c : keySlice.getColumns()) {
                  if (c.getColumn() != null) {
                      String name = new String(c.getColumn().getName(),
                              ENCODING);
                      String value = new String(c.getColumn().getValue(),
                              ENCODING);
                      // long timestamp = c.getColumn().getTimestamp();
                      if (name.equals("name")) {
                          group.setName(value);
                      }
                  }
              }

              list.add(group);
          }
          return list;

      } catch (Exception exception) {
          exception.printStackTrace();
      }
      return null;
  }

  @Override
  public Group findById(Integer id) {
      try {
          SlicePredicate slicePredicate = new SlicePredicate();
          SliceRange sliceRange = new SliceRange();
          sliceRange.setStart(new byte[] {});
          sliceRange.setFinish(new byte[] {});
          slicePredicate.setSlice_range(sliceRange);

          ColumnParent columnParent = new ColumnParent(COLUMN_FAMILY);
          List<ColumnOrSuperColumn> result = client.get_slice(KEYSPACE, id
                  .toString(), columnParent, slicePredicate,
                  ConsistencyLevel.ONE);

          if (result == null || result.isEmpty())
              return null;

          Group group = new Group();
          group.setId(id);

          for (ColumnOrSuperColumn c : result) {
              if (c.getColumn() != null) {
                  String name = new String(c.getColumn().getName(), ENCODING);
                  String value = new String(c.getColumn().getValue(), ENCODING);
                  // long timestamp = c.getColumn().getTimestamp();
                  if (name.equals("name")) {
                      group.setName(value);
                  }
              }
          }

          return group;
      } catch (Exception exception) {
          exception.printStackTrace();
      }
      return null;
  }

  @Override
  public void save(Group group) {
      try {
          long timestamp = System.currentTimeMillis();
          Map<String, List<ColumnOrSuperColumn>> job = new HashMap<String, List<ColumnOrSuperColumn>>();

          List<ColumnOrSuperColumn> columns = new ArrayList<ColumnOrSuperColumn>();
          Column column = new Column("name".getBytes(ENCODING), group
                  .getName().getBytes(ENCODING), timestamp);
          ColumnOrSuperColumn columnOrSuperColumn = new ColumnOrSuperColumn();
          columnOrSuperColumn.setColumn(column);
          columns.add(columnOrSuperColumn);

          job.put(COLUMN_FAMILY, columns);

          client.batch_insert(KEYSPACE, group.getId().toString(), job,
                  ConsistencyLevel.ALL);
      } catch (Exception exception) {
          exception.printStackTrace();
      }
  }

  @Override
  public void remove(Group group) {
      try {
          ColumnPath columnPath = new ColumnPath(COLUMN_FAMILY);
          client.remove(KEYSPACE, group.getId().toString(), columnPath,
                  System.currentTimeMillis(), ConsistencyLevel.ALL);
      } catch (Exception exception) {
          exception.printStackTrace();
      }
  }

}

Now make sure you project is fully "buildable" and make the testing class run! Unless your Cassandra server is offline, the resulting on JUnit is "evergreen".

Here are the tasks supposed to be performed by the given unitary test:

instantiate a group identified by the key 123 and persist it, searching it and comparing to the expected object;
persist a group, remove it on the database and then try to search for its key expecting not to find it;
insert 10 instances of groups with keys numbered from 12301 to 12310 and test their retrieval.

If you're so skeptical as me, you'll love to search for those persisted data by hands. In that case, prompt to the Cassandra's client and try sending some commands to retrieve groups and their columns recently stored. Here are some suggestions:


cassandra> count ContactList.Groups['12301']
1 columns

cassandra> get ContactList.Groups['12301']
=> (column=name, value=Test Group 1, timestamp=1283287745613)
Returned 1 results.

cassandra> get ContactList.Groups['12301']['name']
=> (column=name, value=Test Group 1, timestamp=1283287745613)

Conclusion

We have reached the end of this study, at last! In this article some concepts about a NoSQL-based database called Cassandra were introduced. Besides, we implemented in Java a very simple example of persistence into a Cassandra keyspace using a low-level API named Thrift. In the end we just searched for the stored values in the database through Cassandra's client.

References

[1] Apache Cassandra
[2] Getting Started with Cassandra
[3] JUnit - Unit Testing Framework
[4] Apache Thrift
[5] Cassandra Wiki - Thrift Examples
[6] Cassandra Wiki - Data Model
[7] A Quick Introduction to the Cassandra Data Model

Configurando Bash Completion no Debian

2010-02-10T07:55:00.000-08:00

O bash completion é uma poderosa ferramenta para administradores Linux que utilizam o shell para realizar suas tarefas, permitindo com que comandos, caminhos de arquivos e mesmo parâmetros específicos sejam autocompletados. Em geral essa feature é habilitada por padrão no Linux, porém em algumas distribuições, tal como no Debian, é preciso configurá-la manualmente. Este é justamente o tema do artigo em questão.

Antes de qualquer coisa, certifique-se de que o pacote bash-completion esteja instalado, executando a instrução a seguir:


# apt-get install bash-completion

Em seguida, é preciso indicar a utilização do bash-completion no perfil do bash. Isso pode ser feito individualmente para cada usuário, através do arquivo ~/.bash_profile, ou para todo o sistema. Para essa segunda opção, modifique o arquivo /etc/bash.bashrc, usando um editor qualquer:


# vi /etc/bash.bashrc

Verifique se as seguintes linhas estão incluídas no arquivo de configurações. Inicialmente elas já existem, porém estão comentadas (i.e. iniciadas com o caracter "#"):


# enable bash completion in interactive shells
if [ -f /etc/bash_completion ] && ! shopt -oq posix; then
  . /etc/bash_completion
fi

A última etapa consiste em testar se o mecanismo de autocomplemento está funcionando. Para isso, é preciso sair da sessão (i.e. fazer um log out do terminal) e iniciar um novo login. Feito isso, tente digitar a inicial de algum comando (ex: service) e em seguida pressionar a tecla TAB uma vez:


# serv<TAB>

Agora que o comando "service" foi autocompletado, pressione a tecla TAB mais duas vezes para que a respectiva lista de seus possíveis parâmetros seja exibida na tela:


# service <TAB>
hwclock               networking            skeleton              udevtrigger
apache2               hwclock-save          ondemand              umountfs
bootlogd              keyboard-setup        postfix               rsyslog
console-setup         procps                udev                  umountroot
dmesg                 rc                    sendsigs              urandom
halt                  nagios3               rc.local              single

Caso o comando possua múltiplos argumentos, como é o caso do "service", podemos utilizar o TAB mais uma vez, o que fará com que apenas instruções que façam sentido sejam listadas:


# service nagios3 <TAB>
force-reload  reload        restart       start         status        stop

Agora é só digitar a inicial do argumento, pressionar TAB mais uma vez e executar de vez a instrução completa:


# service nagios3 status
* checking /usr/sbin/nagios3...                                      [OK]

Pronto! Bem fácil, né?

Reading stdin on scripts for handling SNMP traps

2010-02-05T05:25:00.000-08:00

In a shell, sometimes data are read through standard input (i.e STDIN) rather than arguments, which are most commonly used. For that reading purpose there is an instruction named "read", whose syntax is exemplified below:


$ read var
123

$ echo "var: $var"
var: 123

Today I was wondering how could I retrieve all values passed to a given Shell script considering that one does not know how many items are available. There are a lot of script samples of reading regular files, line by line, using "while" loop along with "read" command. But what if one needs to read the standard input?

Fortunately, there is a special device on Linux called /dev/stdin, which promised to give the expected outcome. And here's a testing script based on that assertion:


#!/bin/bash
while read s
do
    echo "s = $s"
done < /dev/stdin

And it works perfectly! Well, but.. is there any real application for such trick?

Yes, indeed! My real motivation was to debug and understand why the integration between Net-SNMP (i.e. snmptrapd service) and ZABBIX was not working when SNMP traps were received by the server. Those traps were supposed to be forwarded by snmptrapd to a Shell script called snmptrap.sh, which invoked zabbix_sender along with the received arguments.

Thus, in order to fine-grained debbuging, I created another script, named snmptrapdebug.sh with the contents below:


#!/bin/bash

echo "Request received: $0"
echo "--------------------------------------------------"

while read param
do
    echo -e "$param "
done < /dev/stdin

echo "--------------------------------------------------"
echo

exit 0

On the configuration file snmptrapd.conf the following line was appended:


traphandle default /bin/bash /home/zabbix/snmptrapdebug.sh

In order to send a simple SNMP trap the following command was issued:


$ snmptrap -v 1 -c public localhost .1.3.6.1.4.1.24.0 localhost 6 12345678 8 .1.3.6.1.4.1.24.12.10.22.64 s "2010-02-04 17:03:18,352,DEBUG,main,SimpleTest,This is a Debug Message"

So, here are the LOG results after receiving a SNMP trap by the server:


2010-02-05 11:48:32 localhost [127.0.0.1] (via UDP: [127.0.0.1]:40338) TRAP, SNMP v1, community public
SNMPv2-SMI::enterprises.24.0 Enterprise Specific Trap (12345678) Uptime: 0:00:00.08
SNMPv2-SMI::enterprises.24.12.10.22.64 = STRING: "2010-02-04 17:03:18,352,DEBUG,main,SimpleTest,This is a Debug Message"
Request received: /home/zabbix/snmptrapdebug.sh
--------------------------------------------------
localhost
UDP: [127.0.0.1]:40338
DISMAN-EVENT-MIB::sysUpTimeInstance 0:0:00:00.08
SNMPv2-MIB::snmpTrapOID.0 SNMPv2-SMI::enterprises.24.0.0.12345678
SNMPv2-SMI::enterprises.24.12.10.22.64 "2010-02-04 17:03:18,352,DEBUG,main,SimpleTest,This is a Debug Message"
SNMP-COMMUNITY-MIB::snmpTrapAddress.0 127.0.0.1
SNMP-COMMUNITY-MIB::snmpTrapCommunity.0 "public"
SNMPv2-MIB::snmpTrapEnterprise.0 SNMPv2-SMI::enterprises.24.0
--------------------------------------------------

That's it! I hope that tip is worthful for someone else. :)

References:

Pastis: orgulho do sul da França

2010-01-31T23:34:00.000-08:00

Para abrir o apetite ou simplesmente para degustar, nada como um bom pastis, uma das bebidas mais apreciadas na França, especialmente no sul do país. Trata-se de um licor a base de anis, de coloração clara e que se torna cáqui com a adição de água.

O pastis foi criado na cidade de Marselha como um substituto para o absinto, o qual havia sido banido pelo governo francês em 1915. O absinto era uma bebida popular na época, e a sua proibição levou as pessoas a experimentarem diferentes combinações de ervas e álcool em um esforço para produzir uma bebida que não levava o seu ingrediente banido, o funcho (anis-doce). Mais anis e açúcar foram adicionados a uma menor concentração de álcool e assim surgiu o pastis. Um desse destiladores de fundo de quintal foi Paul Ricard. Ele levava suas misturas a bares ao redor da cidade pedindo para as pessoas a experimentarem a fim de que a sua respectiva fórmula pudesse ser aperfeiçoada. Ele iniciou a produção comercial do Ricard em 1932, conhecido sobretudo pelo slogan "Le Vrai Pastis de Marseille" (o Verdadeiro Pastis de Marselha).

Em geral devemos colocar cinco partes de água para cada parte de pastis, não esquecendo do gelo. Devo lembrar que uma bebida semelhante ao pastis supostamente embriagou van Gogh quando ele cortou a orelha...

O pastis permanece como um dos mais populares aperitivos na França, além de acompanhar muito bem pratos como peixes, frutos do mar, porco e frango.

Pastis 51 (número ironicamente igual à nossa "caninha" de Pirassununga) é a marca líder na França, porém Ricard e Pernod são provavelmente as marcas mais conhecidas fora do país. Ao pedir um pastis em um bar ou restaurante, simplesmente diga "Du Ricard, s.v.p !". Ou faça como Serge Gainsbourg e peça um "102" (i.e. um 51 duplo).. :D

Referências:
http://fr.wikipedia.org/wiki/Pastis

PostgreSQL monitoring on ZABBIX

2009-12-07T11:39:00.000-08:00

I've recently started to get to know ZABBIX [1] a little deeper, especially regarding PostgreSQL database servers monitoring. At first sight I thought it an incredible monitoring and notification system as it fulfills most of the requirements I wondered to have in Cedrus [2].

After setting up some basic OS-related items to be monitored, I was searching for PostgreSQL specific configurations and then I found a wiki [3] on ZABBIX UserParameters. It is indeed very simple! You just need to create SQL statements and then invoke them with psql. If successful, you could append the corresponding lines into ZABBIX agent configuration file and restart it. Then, in the front-end application, what is left to do is to properly set these PostgreSQL parameters to a given host.

I've created some additional parameters I always use in PostgreSQL instances in order to monitor the server health. In this case, I previously created a user called "zabbix" and a database with same name on PostgreSQL.

Here are the included lines on /etc/zabbix/zabbix_agentd.conf:


# PostgreSQL custom parameters

# instance version
UserParameter=pgsql.version,psql -U zabbix zabbix -Atc 'select version()'

# instance databases summary
UserParameter=pgsql.db.summary,psql -c "select a.datname, pg_size_pretty(pg_database_size(a.datid)) as size, cast(blks_hit/(blks_read+blks_hit+0.000001)*100.0 as numeric(5,2)) as cache, cast(xact_commit/(xact_rollback+xact_commit+0.000001)*100.0 as numeric(5,2)) as success from pg_stat_database a order by a.datname"

# total databases size
UserParameter=pgsql.db.totalsize,psql -Atc "select sum(pg_database_size(datid)) as total_size from pg_stat_database"

# specific database size (in bytes)
UserParameter=pgsql.db.size[*],psql -Atc "select pg_database_size('$1') as size"

# database cache hit ratio (percentage)
UserParameter=pgsql.db.cache[*],psql -Atc "select cast(blks_hit/(blks_read+blks_hit+0.000001)*100.0 as numeric(5,2)) as cache from pg_stat_database where datname = '$1'"

# database success rate (percentage)
UserParameter=pgsql.db.success[*],psql -Atc "select cast(xact_commit/(xact_rollback+xact_commit+0.000001)*100.0 as numeric(5,2)) as success from pg_stat_database where datname = '$1'"

After restarting ZABBIX Agent, you can check whether the added parameters are valid by issuing zabbix_get command in a shell. For instance, to query PostgreSQL instance version, type this:


$ zabbix_get -s localhost -k pgsql.version
PostgreSQL 8.3.3 on i486-pc-linux-gnu, compiled by GCC cc (GCC) 4.2.3 (Ubuntu 4.2.3-2ubuntu7)

This other script returns an overview of existing databases in the instance, highlighting their names, size in disk, cache hit ratio and percentage of successful transactions:


$ zabbix_get -s localhost -k pgsql.db.summary
   datname   |  size   | cache | success 
-------------+---------+-------+---------
 auction5    | 4400 kB |  0.00 |    0.00
 auditing    | 4512 kB |  0.00 |    0.00
 escola      | 4656 kB |  0.00 |    0.00
 postgres    | 4223 kB | 99.81 |  100.00
 rodrigo     | 4144 kB |  0.00 |    0.00
 template0   | 4144 kB |  0.00 |    0.00
 template1   | 4144 kB |  0.00 |    0.00
 zabbix      | 10 MB   | 99.99 |  100.00
 zahle       | 86 MB   |  0.00 |    0.00
(9 registros)

In order to measure total space in disk occupied by the entire instance, this script should be used:


$ zabbix_get -s localhost -k pgsql.db.totalsize
1507326452

On the other hand, to retrieve the space (in bytes) occupied by a single database, you just need to specify its name in the parameter key, as exemplified below:


$ zabbix_get -s localhost -k pgsql.db.size[auction5] 
4505604

Likewise, to recover cache hit ratio (in percentage) of a single database, use this key:


$ zabbix_get -s localhost -k pgsql.db.cache[zabbix]
99.99

The percentage of successful transactions in relation to all attempts is given by this parameter:


$ zabbix_get -s localhost -k pgsql.db.success[postgres]
100.00

After testing these parameters, it is time to set them up onto ZABBIX Frontend as illustrated below:

It is very interesting to further add some traps and actions based on the configured items. For example, to send an email to the DBA every time a given database grows up faster than expected or whether its cache ratio starts to lower significantly.

At Joe Uhl's blog there is a post [4] concerning using ZABBIX to monitor PostgreSQL TPS (Transactions per Second). It is an interesting source as it explains how to configure deltas and graphs in ZABBIX.

References:

[1] ZABBIX Monitoring Solution
[2] Cedrus: PostgreSQL Manager
[3] ZABBIX Wiki - PostgreSQL UserParameters
[4] Monitoring PostgreSQL TPS with Zabbix

Creating Demoiselle JPA-compliant DAOs

2009-12-01T04:51:00.000-08:00

This article aims at providing a walk through on building Data Access Object (DAO) [1] components using Java Persistence API (JPA 1.0) [2], since the latter is now fully supported in the 1.1 release of Demoiselle Framework [3].

The reference source codes used in the following sections were retrieved from Auction5 [4], a sample web application specifically designed to exemplify JPA usage along with Demoiselle.

i. Introducing support elements

The main characters on the new Demoiselle persistence handling mechanism are IDAO [5] and JPAGenericDAO [6].

IDAO is a simple interface containing data manipulation and retrieval methods. It is, above all, a markup interface intended to be implement by whoever class playing a DAO role. Therefore, no matter whether the final class will handle objects from a DBMS or an XML file, using pure JDBC, Hibernate or JPA, it must necessarily implement IDAO.

On the other hand, JPAGenericDAO is an abstract class intented to leverage productivity by providing the derived class with a bunch of persistence-related ready-to-use methods such as: findById(), findByJPQL(), insert(), update(), and remove().

ii. Defining entities

Entity classes are usually the first elements to be created on a Java application powered by JPA persistence.

In the class diagram below the whole application entities considered in the examples are depicted.

iii. Designing interfaces

In order to build DAO artifacts on Demoiselle, one needs to previously create their respective interfaces. Besides, these interfaces must extend IDAO by providing the targeted entity class in the generics brackets (e.g. <Auction>).

The signatures of additional needed methods must be indicated on each interface, remembering that all methods provided by IDAO are already implicitly included.

For instance, for the Auction entity, there must be created IAuctionDAO interface by extending IDAO and declaring additional listing methods. Its final code is shown below:


public interface IAuctionDAO extends IDAO<Auction> {
    public Auction findById(Long id);
    public List<Auction> listOpenAuctionsByCategory(Category category);
    public List<Auction> listNewestAuctions(int quantity);
    public List<Auction> listMostOfferedAuctions(int quantity);
    public List<Auction> listEndingSoonAuctions(int seconds, int quantity);
    public List<Auction> listCheapestPriceAuctions(int quantity);
    public List<Auction> listOpenAuctionsByItem(Item item);
    public List<Auction> listOpenEndedAuctions(Date timestamp);
}

Likewise, interfaces for Bid, Category, Item, and Order entities suggest the structures below:


public interface IBidDAO extends IDAO<Bid> {
    public List<Bid> listLastBidsForAuction(Auction auction, int quantity);
}


public interface ICategoryDAO extends IDAO<Category> {
    public Category findById(Short id);
    public List<Category> listAvailableCategories();
    public List<Category> listAllCategories();
}


public interface IItemDAO extends IDAO<Item> {
    public Item findById(Integer id);
    public List<Item> listByCategory(Category category);
}


public interface IOrderDAO extends IDAO<Order> {
    public Order findById(Long id);
    public List<Order> listOrdersByLogin(String login);
}

On an application powered by Demoiselle all those interfaces are usually placed on a subpackage named "persistence.dao" inside the main package tree (e.g. "br.gov.demoiselle.sample.auction5.persistence.dao").

iv. Designing implementation classes

After creating the DAO interfaces, the next step is to build their respective implementation classes. For each interface, there must be created a concrete class declared with a reference to that interface.

In addition, in order to boost development time and yet help the developer with several coding facilities, these new classes should extend JPAGenericDAO superclass described previously.

Thus, DAO implementation classes should contain the declarations below:


public class AuctionDAO extends JPAGenericDAO<Auction> implements IAuctionDAO { ... }

public class BidDAO extends JPAGenericDAO<Bid> implements IBidDAO { ... }

public class CategoryDAO extends JPAGenericDAO<Category> implements ICategoryDAO { ... }

public class ItemDAO extends JPAGenericDAO<Item> implements IItemDAO { ... }

public class OrderDAO extends JPAGenericDAO<Order> implements IOrderDAO { ... }

As a result of "implements", every method declared in an interface must be implemented and fulfilled with actual code by the developer in the class. However, there are exceptions: methods belonging to IDAO are already implemented in JPAGenericDAO, and therefore do not need to be reimplemented.

Demoiselle Framework suggests hosting these classes onto a subpackage named "implementation" right inside the subpackage containing the interfaces (e.g. "br.gov.demoiselle.sample.auction5.persistence.dao.implementation").

v. Implementing classes methods

In the next sections there will be explored several ways of implementing the methods declared in the interfaces, by using existing code, by issuing JPQL queries and even through SQL native queries.

1. Superclass methods

The first alternative on building persistence methods in the DAO is the simplest one: to directly invoke existing methods of JPAGenericDAO superclass.

For example, a listAllCategories() method designed to bring all categories could be coded in CategoryDAO using the following contents:


public List<Category> listAllCategories() {
    return findAll();
}

If the method name is to be kept, one should use the "super" reference variable to prevent infinite loops, as illustrated below in findById() method for CategoryDAO:


public Category findById(Short id) {
    return super.findById(id);
}

2. JPQL fixed string

As an evolution to EJB QL, JPA 1.0 introduces a query language called JPQL (Java Persistence Query Language). This language's synthax is similar to Hibernate's HQL and offers the same purpose: to send instructions to the DBMS by using a language closer to object-oriented world.

It is made available to the JPAGenericDAO class a method called findByJPQL(), which internally creates a JPQL query according to a given string, executes the query, builds a list of entities and finally returns the latter in a type-safe fashion.

When coding a DAO method on Demoiselle, we could use fixed JPQL queries whenever there is no need to give parameters to them. The CategoryDAO implementation class exhibits the usage of fixed string JPQL queries through findByJPQL():


public List<category> listAvailableCategories() {
    return findByJPQL("select c from Category c where c.active = true order by c.name");
}

3. JPQL with positional parameters

As most of our real-world queries involve passing parameters to them, JPQL also offers the possibility of doing this. Actually, there are two ways of passing parameters to a query: using positional (numbered) or named parameters.

Positional parameters will be explored in the current section. They are expressed in the JPQL string with the "?" character followed by an integer number starting from 1.

On JPAGenericDAO the method findByJPQL() is overloaded so that it can handle positional parameters as well. The AuctionDAO implementation class exhibits the usage of JPQL queries with positional parameters on listAllAuctionsByItem() method:


public List<Auction> listAllAuctionsByItem(Item item) {
 
    String jpql = "select a from Auction a where a.item = ?1";
 
    List<Auction> result = findByJPQL(jpql, item);
 
    return result;
}

Demoiselle Framework introduces pagination mechanisms through Page and PagedResult classes. In order to use them, one needs to invoke an other overloaded version of findByJPQL(), which accepts a Page object as the second argument. The BidDAO implementation class comes with listLastBidsForAuction() method, which exemplifies pagination of results using the classes described and yet accepts a positional parameter:


public List<Bid> listLastBidsForAuction(Auction auction, int qty) {
  
    Page page = new Page(qty);
  
    String jpql = "select b from Bid b where b.auction.id = ?1 order by b.timestamp desc";
    PagedResult<Bid> pr = findByJPQL(jpql, page, auction.getId());
  
    return (List<Bid>) pr.getResults();
}

4. JPQL with named parameters

There are some situations where numbered parameters in a query might be a confusing or not preferred alternative. In order to handle it, JPA specification foresees named parameters on JPQL queries.

According to JPA, a named parameter in JPQL is indicated with the ":" character followed by an unique string which specifies it.

Inside Demoiselle, using named parameters consists of a map instantiation and the execution of another overloaded findByJPQL(), this time receiving in addition a Map object as argument. The AuctionDAO implementation class exhibits the usage of JPQL queries with bound named parameters:


public List<Auction> listOpenAuctionsByCategory(Category category) {
 
    String jpql =
        "select a from Auction a " +
        "where a.status = :status " +
        "  and a.deadline > :deadline " +
        "  and a.item.category = :category " +
        "order by a.item.description";
 
    Map<String, Object> params = new HashMap<String, Object<();
    params.put("status", Status.OPEN);
    params.put("deadline", new Date());
    params.put("category", category);
 
    List<Auction> result = findByJPQL(jpql, params);
 
    return result;
}

5. Named JPQL query

As an alternative introduced by Hibernate, JPA brings a feature called named queries, which aims at writing complex JPQL statements inside entity classes in the form of annotations. These so-called annotated queries are then referenced back when issuing JPQL queries.

JPAGenericDAO comes handy once again by providing another method: findByNamedQuery(). This method receives a string containing the name of a JPQL query whose statement is already on an entity. For instance, let the Item entity with a @NamedQuery annotation right above its class declaration:


@NamedQuery(
    name = "itemsByCategory",
    query = "select i from Item i where i.category.id = ?1 order by i.description"
)
public class Item implements IPojoExtension { ... }

Then, the respective ItemDAO implementation class should make use of JPAGenericDAO's findByNamedQuery() as shown in the code belonging to listByCategory() method:


public List<Item> listByCategory(Category category) {
    return findByNamedQuery("itemsByCategory", category.getId());
}

Note that in the previous example a positional parameter was placed. But what if we need named parameters alongside named queries? The answer is: an overloaded version of findByNamedQuery() might take place! For example, the Auction entity presents a @NamedQuery according to the code below:


@NamedQuery(
    name = "openAuctionsByItem",
    query = "select a from Auction a where a.status = :status and a.deadline > :deadline and a.item = :item"
)
public class Auction implements IPojoExtension { ... }

Observe that named parameters on a named JPQL query are indicated in the annotation as if they were to be used by findByJPQL() method. And then, in the same manner, we need to create a map object in order to invoke the method, which this time is findByNamedQuery(). Take a look at the contents of AuctionDAO, particularly the listOpenAuctionsByItem() method:


public List>Auction< listOpenAuctionsByItem(Item item) {
 
    Map<String, Object> params = new HashMap<String, Object<();
    params.put("status", Status.OPEN);
    params.put("deadline", new Date());
    params.put("item", item);
 
    List<Auction> result = findByNamedQuery("openAuctionsByItem", params);
 
    return result;
}

6. Native SQL query

So, are we constrained to JPQL statements and object-oriented queries when adopting JPA persistence mechanism on a Java application? Not at all!

There are some cases where performance matters most of everything and the only possibility of achieving it is by executing native SQL instructions, usually made through JDBC in the Java world. This particular approach can boost up SQL execution time, especially for those who prefer to create their own database stored procedures or functions.

Despite this development behavior can lead to a given DBMS dependency and thus compromising application portability, the architect still might choose to create native SQL queries. JPA specification was developed with a rich set of components (i.e. interfaces and annotations) concerning the invocation of native SQL instructions.

In Demoiselle, once again, JPAGenericDAO can help the developer with a pre-built method: findByNativeQuery(). Take the example of the listMostOfferedAuctions() method inside AuctionDAO, which contains SQL directed to PostgreSQL DBMS:


public List<Auction> listMostOfferedAuctions(int quantity) {
 
    String sql =
        "select d.* from ( " +
        "  select b.auction_id, count(b.id) " +
        "  from bids b " +
        "  inner join auctions a on (a.id = b.auction_id) " +
        "  where a.status = ?1 and a.mode <> ?2 " +
        "  group by b.auction_id " +
        "  order by count(b.id) desc " +
        "  limit ?3) c " +
        "join auctions d on (d.id = c.auction_id) " +
        "where d.bestbid_id is not null " +
        "order by c.count desc";
 
    List<Auction> result = findByNativeQuery(sql,
         Status.OPEN.ordinal(), Mode.SELLING.ordinal(), quantity);

    return result;
}

Another method of AuctionDAO, listEndingSoonAuctions(), makes use of native SQL query and positional parameters:


public List<Auction> listEndingSoonAuctions(int seconds, int quantity) {
 
    String sql =
        "select * " +
        "from auctions " +
        "where status = ?1 " +
        "  and deadline - current_timestamp between '0 s' and cast(?2 as interval) " +
        "order by deadline " +
        "limit ?3";
 
    List<Auction> result = findByNativeQuery(sql,
         Status.OPEN.ordinal(), seconds + " s", quantity);

    return result;
}

But now another issue is raised: what if we need native SQL query along with named parameters? The answer might be simple: just merge both techniques into one, right? Well, partially.

According to JPA 1.0 specification, a named parameter is an identifier that is prefixed by the ":" symbol, and its use applies just to JPQL, therefore not being defined for native queries. Thus, only positional parameter binding may be portably used for native queries.

Despite that lack of portability, JPA implementations usually offer named parameters on native queries. Unfortunately there is no common sense yet, so that implementations consider "?" or ":" symbols when defining named parameters in a native SQL string. This is better illustrated in the code below, taken from AuctionDAO's listCheapestPriceAuctions() method, which is prepared to run on Hibernate EntityManager JPA implementation:


public List<Auction> listCheapestPriceAuctions(int quantity) {
 
    // WARNING: prepared for Hibernate - use "?" instead of ":" for TopLink or EclipseLink
    String sql =
        "select d.* from (" +
        "  select a.id, coalesce(b.amount, a.startingprice, a.sellingprice) as price " +
        "  from auctions a left join bids b on (b.id = a.bestbid_id) " +
        "  where a.status = :status " +
        "  order by 2 asc " +
        "  limit :quantity) c " +
        "join auctions d on c.id = d.id " +
        "order by c.price";
 
    Map<String, Object> params = new HashMap<String, Object<();
    params.put("status", Status.OPEN.ordinal());
    params.put("quantity", quantity);
 
    List<Auction> result = findByNativeQuery(sql, params);
 
    return result;
}

Thus, portability is compromised in the previous example, as we need to take special precautions when switching to another JPA implementation, say EclipseLink or Apache OpenJPA.

7. Named native SQL query

As with JPQL, it is possible to store the entire native query statement onto an entity class and then reference it as needed. In order to do that, JPA defines another annotation, @NamedNativeQuery, which has the same purpose of @NamedQuery. After setting the named native query, one could invoke findByNamedQuery() method to execute it and possibly bring back the results. Note that this very same method fits for both JPQL and native SQL queries.

So, consider the named native query "newestAuctions", which is defined on Auction entity file. There, in the @NamedNativeQuery annotation three parameters must be passed: the query unique identifier, the statement itself, and a class indicating the result type. The referenced code is shown below:


@NamedNativeQuery(
    name = "newestAuctions",
    query = "select * from auctions where status = ?1 order by creation desc limit ?2",
    resultClass = Auction.class
)
public class Auction implements IPojoExtension { ... }

At DAO implementation class AuctionDAO this named native query is called inside listNewestAuctions() method, as illustrated in the following code:


public List<Auction> listNewestAuctions(int quantity) {
    List<Auction> result = findByNamedQuery("newestAuctions",
         Status.OPEN.ordinal(), quantity);
}

8. LIQUidFORM tool

JPA 1.0 specification does not provide a Criteria API [7], a feature introduced by Hibernate designed to help developers on building HQL queries. With that we avoid string concatenation and rather we declaratively define the statement, thus reducing the risk of errors in its synthax.

Fortunately JPA 2.0 was conceived with a Criteria API, thus soon JPA implementations will run after that. Meanwhile, a lot of alternatives were developed to fill in such requirement in JPA. Amongst them, one particular tool was employed in the sample code: LIQUidFORM [8].

LIQUidFORM is actually a set of classes and interfaces designed to compose JPQL statements in a type-safe and refactoring proof style. It might be strange at first sight, as it contains methods called select(), from(), and where(), but it is indeed a very interesting idea.

The key to LIQUidFORM is to make use of static imports in the class header, namely the lines below:


import static com.google.code.liquidform.LiquidForm.*;
import static com.google.code.liquidform.Parameters.*;

OrderDAO implementation class exhibits the usage of LIQUidFORM tool for creating a JPQL query inside listOrdersByLogin() method, as shown below:


public List<Order> listOrdersByLogin(String login) {

    Order o = alias(Order.class, "o");

    String jpql =
        select(o).
        from(Order.class).as(o).
        where(eq(o.getLogin(), param(1, String.class))
    ).toString();

    return findByJPQL(jpql, login);
}

vi. Conclusion

In this article we have explored some benefits of building the persistence layer of a Java application by adopting Java Persistence API (JPA 1.0) inside an architecture composed by DAO-based interfaces and implementation classes.

In addition, we have seen the usage of Demoiselle Framework structural IDAO interface and JPAGenericDAO support class when building methods designed to manipulate data through JPA. Those methods also served to exemplify the existing JPA approaches by introducing JPQL and native SQL queries, positional and named parameters, JPQL and SQL named queries, and LIQUidFORM tool.

vii. References

[1] Core J2EE Patterns - Data Access Object
[2] JPA 1.0 (JSR 220)
[3] Demoiselle Framework
[4] Auction5 Application Sample
[5] IDAO interface
[6] JPAGenericDAO class
[7] Criteria API
[8] LIQUidFORM

Roteiro de voo CWB-CGH

2009-11-26T08:35:00.000-08:00

Depois que viciei no M$ Flight Simulator, comecei a prestar atenção nos comandos efetuados pelos pilotos em voos comerciais. A seguir um roteiro de eventos que registrei a partir da janelinha de um A319 de Curitiba (CWB) a São Paulo (CGH)... :) Mapa da rota ao lado.


tempo     evento
--------  ----------------------
19:25:00  entrada (gate 2 CWB)
19:40:15  portas auto
     +30" pushback
19:44:15  throttle up
   45:45  flaps down
     +45" taxi
   55:50  entrada pista 15C
          strobes on
   58:00  alinhamento
      +1' speed up
      +1' pitch up
     +15" gears up
      +1' flaps up 100%
20:02:45  pitch down slowly
   04:15  bank left
     +30" stabilize (NE)
   22:45  início descida
   25:00  avistado litoral SP
   29:30  bank left (N)
   30:00  landing lights on
   31:10  flaps down +1
   37:45  bank right
     +15" stabilize
     +15" flaps down +1
   40:15  bank right
     +20" stabilize
   40:30  bank right
     +20" stabilize
   42:40  bank right
     +20" stabilize
   45:00  throttle down
     +45" slats down
     +10" flaps down +1
     +10" flaps down +1
      +5" pouso autorizado
   47:35  avistado Tietê
   49:45  avistada Av. Bandeirantes
   50:15  tocada pista 17R CGH
      +5" spoilers/reversos acionados
     +30" saída pista principal
     +10" taxi
     +20" flaps/spoilers recolhidos

PS: Sim, não há registro do abaixamento do trem de pouso. Simplesmente porque não percebi quando ele ocorreu... :(

Seattle-Vancouver on TAM's A319

2009-11-18T18:45:00.000-08:00

From Seattle-Tacoma International Airport [1] to Vancouver International Airport [2] on a TAM Airlines [3] Airbus A319 [4] aircraft... :D

References:
[1] http://en.wikipedia.org/wiki/Seattle-Tacoma_International_Airport
[2] http://en.wikipedia.org/wiki/Vancouver_International_Airport
[3] http://en.wikipedia.org/wiki/TAM_Airlines
[4] http://en.wikipedia.org/wiki/Airbus_A319#A319

Unicode to Java string literal converter

2009-11-16T12:41:00.000-08:00

I was having some headaches writing properties files valid across several platforms due to encoding issues. However, by googling around today I found this very interesting JavaScript-based conversion tool:

http://www.snible.org/java2/uni2java.html

Thanks to Ed Snible!

Resumos das palestras do TDC 2009

2009-11-10T05:52:00.000-08:00

Seguem resumos das palestras dos keynotes no The Developer's Conference 2009 - Floripa [1] ministradas no dia 9 de novembro.

Tendências em Java EE: Como serão os próximos 5 anos - Rod Johnson

O criador do Spring [2], um dos mais populares frameworks em Java, apresentou nesta palestra suas previsões para o futuro da tecnologia da informação como um todo. Nessa linha, afirmou que "estamos a ponto de presenciar a maior ruptura em TI desde o declínio do mainframe", impulsionada por forças como Cloud Computing, Lei de Moore e o fato de os custos estarem se voltando para as pessoas em contraposição ao hardware.

Disse também que sistemas gerenciadores de bancos de dados relacionais (SGBDRs) não irão embora, e ao invés disso haverá grande ascensão da chamada "persistência poliglota", onde haverão diferentes meios de se armazenar os dados. (Aqui entra a JPA!)

Sobre a computação em nuvem, afirmou que ela está mais próxima que SOA e que possui as seguintes vantagens características: ser abstrata, integrada e opinativa, dinamicamente escalável e consumida como um serviço, além de inevitável devido a questões econômico-ambientais. E com isso, disse que nós desenvolvedores nos tornaremos mais poderosos! :) Precisaremos fornecer APIs direcionadas para a nuvem...

Com relação aos frameworks, iniciou com a frase "Frameworks can't be too disruptive!", alegando que o código aberto possibilitou a criação de excelentes soluções de software, mas por outro lado fez do Java uma plataforma confusa para os iniciantes.

Finalizou dizendo que nessa era de desordem a abstração será mais importante do que nunca, citando como possíveis gerenciadoras para esse caos no software as seguintes soluções: Grails, Spring, Roo e Rails, além de especificações como a JPA.

Java EE 6 - Uma grande evolução - Mike Keith

O Mike é o líder de importantes especificações do Java, tal como EJB 3 e JPA, e iniciou esta palestra apresentando os objetivos e destaques para a esperada versão 6 do Java EE [3], especialmente JSF 2.0, Servlet 3.0, EJB 3.1 e JPA 2.0.

Em seguida resumiu cada uma das novas funcionalidades: Java Profiles, JNDI Namespaces, Data Source Definition, novas anotações para Web (ex: @WebServlet, @ServletFilter, @WebServletContextListener), Web Fragments, Async Processing (mesmo para session beans!). Além disso, para o EJB 3.1 teremos: interfaces opcionais, singleton, concorrência, timer estilo cron, EJB lite, contêiner embutido e novas regras de empacotamento.

A JPA 2.0 terá cache compartilhado, melhorias na JPQL e a inclusão de uma API de Criteria. Os Managed Beans substituirão os MBs atuais do JSF e haverão anotações para injeção comuns a JSE e JEE, além do início de sua padronização. Os WebBeans serão contemplados pela CDI 1.0, a qual basicamente visará gerenciar objetos injetáveis ligados a contextos do ciclo de vida, possuindo anotações como @Inject, @Qualifier, @Scope, @Named e @Singleton.

Outra questão importante refere-se ao processo de limpeza, em que determinadas tecnologias tornaram-se obsoletas e entrarão no estágio de poda (pruning stage). Entre elas citou JAX-RPC, EJB Entity Beans, JAXR e JSR-88 Deployment. Os fornecedores não serão obrigados a implementá-las em releases futuras.

Por fim, a notícia que mais aguardávamos, a data do lançamento. Mike disse que os grupos de experts das JSR praticamente já concluíram seus trabalhos e que a Java EE 6 será liberada no dia 10 de dezembro!

JSF 2.0: Uma Abordagem Completa - Ed Burns

Ed iniciou a palestra fazendo uma pequena introdução sobre a tecnologia JavaServer Faces a fim de balizar o conhecimento do público. Apresentou o ciclo de vida no processamento de requisições do JSF e as funcionalidades desejadas (conversão e validação de dados, fluxo de páginas, integração com banco de dados, desempenho, segurança, etc).

Enfatizando os 4 pilares do JSF 2.0 (visão, interação com o modelo, navegação e ciclo de vida), destacou que as visões passarão a ser páginas Facelets, haverá a anotação @ManagedBean, e que o JSP será depreciado.

A Expression Language será mais poderosa, e na navegação de páginas o uso de XMLs será opcional, devido à nova navegação implícita, altamente dinâmica e especialmente útil com os novos objetos do tipo "flash". Com relação ao ciclo de vida, haverá um excelente suporte para o uso de Ajax, além do advento dos System Events, finos ouvintes dos eventos do ciclo.

Spring Roo - Rod Johnson

Em substituição ao Chris Schalk da Google, Rod fez uma palestra prática totalmente sem slides apresentando uma nova solução da SpringSource: o Spring Roo [5].

Trata-se de um gerador de código em Java que como qualquer do gênero visa impulsionar a produtividade do desenvolvimento. Achei muito semelhante ao Ruby on Rails: um prompt de comando (shell) que executa diversas tarefas, especialmente a criação automática de inúmeros arquivos e o fornecimento de um servidor Web simples para os testes. Funciona até auto-completion no estilo bash!

A instalação do Roo é extremamente simples, mas ele exige que tenhamos o seguinte ambiente: Eclipse IDE com plugins WTP e AJDT. Isso mesmo, o Spring Roo usa extensivamente o AspectJ! São criados arquivos ".aj" a fim de incluir funcionalidade as novas classes.

Outras características importantes foram a criação automática de testes unitários, o fato de o projeto ser gerenciado pelo Maven (são criados diversos arquivos "pom.xml") e a persistência ser manipulada através de JPA.

A mágica do Roo acontece durante o gerenciamento das entidades. Por exemplo, os métodos persist() e findAll() são inseridos em cada entidade através de injeção de aspectos! Achei um tanto bizarro, mas não deixa de ser uma possibilidade interessante. As entidades não precisam declarar métodos getters e setters e nem toString() customizado, pois, novamente, aspectos fazem isso..!

Eis um exemplo de classe criada no Roo:


@Entity
@RooJavaBean
@RooToString
@RooEntity
public class Restaurant {

   private String name;

}

Tal como no Rails, podemos criar com uma agilidade e rapidez descomunal uma aplicação completa incluindo scaffold (i.e. o CRUD), testes unitários, páginas web para apresentação (em JSF!) e toda a configuração da persistência.

Atualmente o Spring Roo está na release candidate 2 e tem previsão de ser liberado no final desse ano. Infelizmente ainda não tem suporte para o NetBeans, devido ao fato de depender do Eclipse AJDT.

Referências:
[1] TDC 2009 Floripa
[2] Spring Framework
[3] Java EE 6 Technologies
[4] JavaServer Faces Technology
[5] Spring Roo

Auction5 sample web application released

2009-11-07T17:25:00.000-08:00

I'm very proud to announce that Auction5, a sample web application built under Demoiselle Framework [1] has just been released to the public. Its complete documentation including source codes retrieval and deployment instructions can be found here: [2].

Auction5 is a MVC-structured application that exemplifies an online auction system using Demoiselle Framework and related technologies. Its main goal is to provide a complete transactional application based on JavaServer Faces (JSF 1.2 [3]) and Java Persistence API (JPA 1.0 [4]) specifications.

Moreover, the sample assembles several top computing technologies such as Java EE, Maven, JBoss Application Server, Apache Tomcat, and PostgreSQL DBMS.

Please send me your comments and or suggestions.

References:
[1] Demoiselle Framework
[2] Auction5 Application Sample
[3] JavaServer Faces 1.2
[4] Java Persistence API 1.0

Script em Shell para monitoração de resultado de concurso

2009-10-28T06:07:00.001-07:00

Eis um script em Shell simples para automatizar o acompanhamento de resultados de concurso pela Internet.

É só configurar a URL desejada e o intervalo entre as verificações (atualmente 15 minutos) e em seguida botar esse negócio para rodar em um terminal. No momento que aparecer alguma novidade o Firefox irá "popar"!


#!/bin/bash

URL="http://www.cops.uel.br/?content=concursos/81_celepar/index.html"

while true
do
 echo "Recuperando página..."
 wget $URL -O res.html
 if [ $? -ne 0 ]; then continue; fi

 echo "Comparando arquivos..."
 if [ -f ult.html ]
 then
  diff res.html ult.html > /dev/null
  dif=$?
 else
  dif=0
 fi

 mv res.html ult.html

 if [ $dif -ne 0 ]
 then
  echo 'Saiu o resultado!'
  firefox $URL
  break
 fi

 echo "Aguardando..."
 sleep 900 # 15 min
done

exit 0

Use-o com moderação. :D

PGCon Brasil 2009

2009-10-07T10:57:00.000-07:00

Pessoal, faltam só alguns dias para o PGCon Brasil 2009, o maior evento de PostgreSQL da América Latina!

Confira maiores informações no site do evento:

[1] http://pgcon.postgresql.org.br/2009/

Poster Motivacional: Fiat Lux!

2009-10-02T04:59:00.000-07:00

Demoiselle JPA in a Nutshell

2009-09-24T11:23:00.000-07:00

One of the greatest new features coming to 1.1 release of Demoiselle Framework [1] is the ability of using JPA [2] (Java Persistence API) in a straightforward manner. JPA specification [3] is a trend in Java development and is often recommended to be used in new applications.

In this article we'll explore the technical steps involved on developing the simplest DAO [4] (Data Access Object) implementation classes with the new JPA support components.

So, what's new in Demoiselle concerning JPA? First of all, there is a new injection aspect towards IDAO [5] interfaces. Basic classes implementing IDAO interface should now simply have an attribute of type EntityManager [6] annotated by @Injection and then with the aid of AspectJ engine and some proxy classes the magic happens!

For instance, consider the following IDepartmentDAO interface in order to manage a Department entity:


public interface IDepartmentDAO extends IDAO<Department> {

    public Department findById(Short id);

}

The respective implementation class DepartmentDAO should have the structure below:


public class DepartmentDAO implements IDepartmentDAO {

    @Injection
    private EntityManager em;

    public Department findById(Short id) {
        return em.find(Department.class, id);
    }

    public boolean exists(Department pojo) {
        return em.contains(pojo);
    }

    public Object insert(Department pojo) {
        em.persist(pojo);
        return pojo;
    }

    public void update(Department pojo) {
        em.merge(pojo);
    }

    public void remove(Department pojo) {
        em.remove(pojo);
    }

}

Simple, isn't it?

EntityManager is the interface used to interact with the persistence context in JPA. A persistence context is a set of entity instances in which for any persistent entity identity there is a unique entity instance. Within the persistence context, the entity instances and their lifecycle are managed. This interface defines the methods that are used to interact with the persistence context. The EntityManager API is used to create and remove persistent entity instances, to find entities by their primary key, and to query over entities.

Within a given unit of work, such as an HTTP request in web applications, a single instance of EntityManager is created and automatically injected on every DAO class that is referenced on that thread. That EntityManager is then closed as the transaction is finished.

Moreover, even outside a complete and robust Java EE application, i.e. without an EJB [3] container, the developer could make easy use of JPA in Demoiselle!

In order to turn it even simpler for the developer, the DAO implementation could still make use of the new JPAGenericDAO [7] support class. Thus, DepartmentDAO could have the following code:


public class DepartmentDAO extends JPAGenericDAO<Department>
    implements IDepartmentDAO {

    public Department findById(Short id) {
        return super.findById(id);
    }

}

In other words, there is no need anymore of storing an EntityManager field and to implement all IDAO methods. That is, there will be almost nothing left to do but customized business rules methods!

If you want to know more about the framework technical details, there will be another article explaining the whole architecture behind the scenes. :D

References:
[1] Demoiselle Framework
[2] The Java Persistence API
[3] EJB 3.0 JPA Specification
[4] Data Access Object
[5] IDAO interface
[6] EntityManager interface
[7] JPAGenericDAO class

Tips on PostgreSQL tuning

2009-09-08T19:42:00.000-07:00

At last I found something more concise towards PostgreSQL DBMS fine tuning. :)

http://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server

Cedrus PostgreSQL Management versão 2.0

2009-09-08T17:38:00.001-07:00

Pessoal, estou organizando esforços para o desenvolvimento da segunda geração do Cedrus [1], um gerenciador para o SGBD PostgreSQL criado em 2006 na CELEPAR e apresentado ao público no CONISLI [4] naquele mesmo ano em São Paulo.

Dessa vez será adotada a plataforma Launchpad para o gerenciamento do projeto. Se você tiver interesse em participar de qualquer das seguintes etapas (requisitos, projeto, desenvolvimento ou testes) ou se desejar somente dar sugestões, mesmo se você só tiver experiência com administração de outros SGBDs, sinta-se livre em conhecer o projeto [2] ou em fazer parte da equipe [3]!

Referências:

[1] Cedrus 1.0 no SourceForge
[2] Cedrus 2.0 no Launchpad
[3] Equipe do Cedrus no Launchpad
[4] CONISLI - Congresso Internacional de Software Livre

Plugins essenciais para o Eclipse...

2009-08-18T07:18:00.000-07:00

Eis alguns plugins interessantes e às vezes indispensáveis para a IDE Eclipse [1]:

m2eclipse [2]: integração do Maven 2 com o Eclipse
JBoss Tools [3]: utilitários para desenvolvimento de projetos Web
AJDT [4]: ferramentas de desenvolvimento com AspectJ (AJ)
Subclipse [5]: suporte ao controle de versionamento Subversion (SVN)

Lembre-se de que o ideal é sempre instalar os plugins via Update Site...

Referências:

[1] http://www.eclipse.org/
[2] http://m2eclipse.sonatype.org/
[3] http://www.jboss.org/tools/
[4] http://www.eclipse.org/ajdt/
[5] http://subclipse.tigris.org/

findjar.com: Encontre o jarro perdido!

2009-08-04T10:43:00.000-07:00

O problema: java.lang.ClassNotFoundException

Como desenvolvedor Java, você com certeza passou por isso alguma vez...
Eis um exemplo de stack trace em que ocorre este tipo de exceção:


Caused by: java.lang.ClassNotFoundException: org.apache.commons.logging.LogFactory
 at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
 at java.security.AccessController.doPrivileged(Native Method)
 at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
 at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
 at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
 ... 21 more

Para resolver isso, é preciso encontrar o arquivo JAR em que a classe referenciada se encontra e colocá-lo no classpath da aplicação que está sendo executada. Mas e se não soubermos onde fica essa classe...?

A solução 1: grep

Se estiver com sorte, você já possui a biblioteca que contém essa classe. Neste caso, no Linux basta executar um rgrep ou um grep -r e copiar o arquivo da biblioteca.

A solução 2: findjar.com

Caso você ainda não possua a biblioteca localmente, é preciso saber o nome do arquivo JAR e onde encontrá-lo na Internet...

Para a primeira pergunta, o site findJAR.com[1] é um grande auxílio. Trata-se de um motor de busca que nos ajuda a encontrar arquivos do tipo JAR que contêm as classes Java requeridas. Basta digitar o nome (simples ou qualificado) da classe que você precisa e iniciar a busca!

findJAR.com é uma mão na roda para resolver com facilidade problemas do tipo NoClassDefFoundError e ClassNotFoundException.

Referências:

[1] http://www.findjar.com

Encoding UTF-8 no Tomcat 6.0

2009-08-02T15:40:00.000-07:00

Ao utilizar o servidor Apache Tomcat 6.0 [1] para o deploy de aplicações Web em Java utilizando o encoding Unicode (UTF-8) [2] podemos enfrentar pequenos problemas quando existe entrada de dados pelo usuário.

Por exemplo, considere a aplicação a seguir que faz uso de um formulário para o cadastro de categorias de produtos:

A anomalia ocorre durante o processo de envio da requisição HTTP do tipo POST [3]. Veja na imagem a seguir o problema na acentuação resultante do formulário:

Uma possível solução para este problema consiste na criação de um filtro, ou Filter [4], um padrão de projeto da especificação Java EE. A função deste filtro é de interceptar as requisições HTTP e forçar a codificação de caracteres para a que for parametrizada (no exemplo, a UTF-8).

A primeira etapa envolve a criação de uma nova classe na aplicação, cujo nome será EncodingFilter e a qual deverá implementar a interface Filter [5] da especificação Java Servlet. Desta forma, o arquivo EncodingFilter.java terá o seguinte conteúdo:


package com.hsiconsultoria.zabumba.filter;

import java.io.IOException;
import javax.servlet.Filter;
import javax.servlet.FilterChain;
import javax.servlet.FilterConfig;
import javax.servlet.ServletException;
import javax.servlet.ServletRequest;
import javax.servlet.ServletResponse;

public class EncodingFilter implements Filter {

  private String encoding;

  public void init(FilterConfig config) throws ServletException {
      this.encoding = config.getInitParameter("encoding");
  }

  public void destroy() {
  }

  public void doFilter(ServletRequest request,
          ServletResponse response, FilterChain chain)
      throws IOException, ServletException {
   
      request.setCharacterEncoding(encoding);
   
      chain.doFilter(request, response);
  }

}

Em seguida é preciso indicar a existência desse novo filtro no arquivo WEB-INF/web.xml, também conhecido como Deployment Descriptor [6]. É neste arquivo que o filtro será devidamente configurado para interceptar as requisições HTTP na aplicação Web.

Sendo assim, basta incluir as linhas seguintes no arquivo web.xml:


  
      EncodingFilter
      EncodingFilter
      
          com.hsiconsultoria.zabumba.filter.EncodingFilter
      
      
          filtro para codificação dos caracteres
          encoding
          UTF-8
      
  
  
      EncodingFilter
      /*

Para finalizar, é só salvar tudo e reinicializar o Tomcat. Veja na tela abaixo o resultado dessa alteração na aplicação:

Referências:
[1] Apache Tomcat 6.0
[2] Formato UTF-8
[3] HTTP/1.1 Method Definitions
[4] Intercepting Filter Pattern
[5] Interface Filter
[6] Java EE Deployment Descriptors

C'est où le son dans Firefox en Ubuntu ?

2009-06-25T11:08:00.000-07:00

Même avec la présence du plugin VLC [1] pour Firefox sur Linux, le son ne marche pas tout simplement...

À travers l'adresse secret "about: config" Firefox, nous avons la liste des plugins installés dans le navigateur, notamment le plugin VLC (qui permet des extensions de fichiers tels que .OGG) :

Néanmois, il nous reste mettre en place le wrapper ALSA [2] pour des applications OSS [3].

Au Linux il existe le paquet "alsa-oss" qui fournisse précisément cette fonctionnalité ! Ainsi, il faut lancer l'instruction suivante en tant que "root" :

# apt-get install alsa-oss

Il vous suffit ensuite de redémarrer Firefox pour que le son passe à marcher !

Je remercie Paulo de Tarso [4] pour ce conseil !

Références :

[1] VLC open-source multimedia framework, player and server

[2] ALSA: the Advanced Linux Sound Architecture

[3] OSS is the free version of the Open Sound System

[4] 70015: A vida com Linux, Mac, Windows e Iphone

Testando HTTP sem usar um navegador

2009-06-19T11:57:00.000-07:00

Um servidor WEB funciona utilizando o protocolo HTTP [1], e responde a requisições tais como "HTTP GET" vindas de um cliente (o browser).

Geralmente não vemos o que está sendo transmitido entre as duas partes: cliente (navegador) e servidor WEB. Entretanto, podemos facilmente inspecionar esse tráfego de dados através da ferramenta telnet [2], disponível em praticamente qualquer plataforma.

Para fazer o teste, primeiro abra um terminal no Linux ou um prompt de comando no Windows. Você vai precisar então digitar o comando com a seguinte sintaxe:

telnet [servidor] [porta]

Por exemplo, para acessar a página do Grupo de Usuários PostgreSQL do Brasil [3], execute o seguinte:

telnet postgresql.org.br 80

Com isso você estará conectado ao servidor WEB especificado e poderá desta forma executar qualquer comando do protocolo HTTP [4] desejado, tal como GET, HEAD ou POST.

Em seguida, para efetuar uma requisição HTTP do tipo GET ao servidor, você precisará executar um comando no seguinte formato:

GET [página] HTTP/1.0

Para exemplificar, faça uma simulação à esta página, digitando o texto a seguir no telnet:

GET /node/61 HTTP/1.0

Atenção: pressione ENTER duas vezes. Se a página existir no servidor, você terá como resultado algo como ilustrado na figura abaixo.

Preste atenção no cabeçalho (header) HTTP enviado pelo servidor após a execução. São justamente estes dados que o browser interpreta automaticamente e o usuário não vê.


HTTP/1.1 200 OK
Date: Sun, 21 Jun 2009 02:08:33 GMT
Server: Apache/2.2.9 (Debian) PHP/5.2.0-8+etch15 mod_ssl/2.2.9 OpenSSL/0.9.8g
X-Powered-By: PHP/5.2.0-8+etch15
Set-Cookie: SESSd41d8cd98f00b204e9800998ecf8427e=baefb8cb6bc0a5ed29340ec05aa6ff38; expires=Tue, 14 Jul 2009 05:41:54 GMT; path=/
Expires: Sun, 19 Nov 1978 05:00:00 GMT
Last-Modified: Sun, 21 Jun 2009 02:08:34 GMT
Cache-Control: store, no-cache, must-revalidate
Cache-Control: post-check=0, pre-check=0
Connection: close
Content-Type: text/html; charset=utf-8

A primeira linha é uma das mais importantes do cabeçalho, pois ela indica se houve sucesso ou erro durante o atendimento da requisição HTTP. O código 200 no exemplo significa que a página HTML foi recuperada com sucesso.

Você pode ainda receber um código de status 404 caso a página não seja encontrada (ou seja, o famoso "Erro 404"), o código 301 se a página tiver sido movida permanentemente ou 401 caso você não tenha autorização para acessá-la.

A lista completa dos códigos de status do protocolo HTTP pode ser consultada aqui [5]. Se você é um administrador de sistemas ou desenvolvedor de aplicações WEB possivelmente esta abordagem lhe será útil. :D

O conteúdo HTML que será renderizado pelo browser vem logo após o cabeçalho.

Referências:
[1] http://pt.wikipedia.org/wiki/Http
[2] http://pt.wikipedia.org/wiki/Telnet
[3] http://postgresql.org.br
[4] Hypertext Transfer Protocol -- HTTP/1.0
[5] RFC 2616 - Status Code Definitions

Sessão nostalgia: Microsoft Windows 3.1

2009-06-14T00:12:00.000-07:00

Ano: 1993. O sistema operacional geralmente encontrado em computadores pessoais aqui era o MS-DOS (versão 5.0, no máximo 6.22!). O Microsoft Windows 3.1, que era opcional na época, não passava de um gerenciador de aplicativos, o "Program Manager". O Linux? Esse não devia ter 2 anos ainda e ninguém fora do meio acadêmico ou talvez longe da Escandinávia tinha conhecimento dele...

O tempo passou e aquele simples agregador de janelinhas foi tomando espaço e quem passou a ser esquecido foi o pobre do MS-DOS, que nos dias atuais não passa de um "Prompt de comando". Ninguém mais ouviu falar de AUTOEXEC.BAT, HIMEM.SYS, GORILLA.BAS, MEMMAKER, QEMM, DOS4GW...

E eis que essa semana vejo um artigo na Internet em que um polonês maluco havia instalado o Microsoft Windows 3.1 no Symbian [1]...! Resolvi seguir o tutorial e fazer o mesmo, só por brincadeira, e funcionou mesmo! Aliás, não serve para nada além disso, pois fica lento, ruim de visualizar na tela, e sem teclado alfanumérico...

Mesmo assim, empolgado pura e simplesmente pelo fator nostálgico, resolvi testar tal inutilidade janélica também no Linux! No caso do N95, o pseudo sistema operacional (AKA Win3.1) estava rodando em cima de um ambiente controlado pelo do DOSBox [2]. Na realidade o maior trabalho nessa façanha foi o porte do DOSBox para o Symbian pelo tal "kolijoco".

No Linux, basta instalar o pacote dosbox. Em distros Debian-like, podemos fazer isso através do comando:

# apt-get install dosbox

Os discos de instalação do Windows 3.1 (na realidade 7 disquetes de 1,44 MB) podemos baixar de fontes extra-oficiais, como sites 4shared ou torrents. Acho que ninguém vai se importar de piratearmos um software que já está debutando, né?

Abra um terminal, dirija-se ao diretório em que o Windows 3.1 será instalado e execute o DOSBox:

$ cd /opt/win31
$ dosbox

Em seguida, já dentro do DOSBox, execute alguns comandos para montar o disco rígido e o primeiro disquete de instalação:

mount c .
mount a /tmp/win31setup

Neste exemplo, os drives serão mapeados no DOS conforme o esquema a seguir:

C: diretório /opt/win31 no Linux;
A: diretório /tmp/win31setup no Linux.

Execute a instalação do Windows através do comando a seguir:

a:\setup.exe

Então basta seguir um clássico "Next, Next, Finish" e ao final executar:

c:
cd \windows
win

No final, eis o resultado: o Microsoft Windows 3.1 sendo executado em uma janelinha de 640x480 pixels no Linux!

Para ter uma idéia do que está acontecendo, veja a tela inteira do Ubuntu Linux, rodando o Gnome como gerenciador de desktop e o DOSBOX ali no canto, totalmente sob controle...

Para facilitar as execuções posteriores, crie um arquivo com o nome "dosbox.conf" dentro do diretório do Windows 3.1 com o seguinte conteúdo:

[sdl]
# sem isso as setas do teclado não funcionam...
usescancodes=false

[autoexec]
# meu teclado é layout alemão, substitua "gr" por "br"
keyb gr
# o resto aqui é para executar o windows ao iniciar
mount c .
c:
cd windows
win

Agora, isso pode parecer piada nos tempos atuais: o Windows 3.1 recém-instalado não chegava a ocupar 12 MB de espaço em disco! Quem imaginaria que um dia o Windows iria exigir 10 GB de HD, hein!

O mais surpreendente é pensar que conseguíamos ser felizes com informática mesmo sem a World Wide Web... Agora vou procurar outras relíquias computacionais... Será que acho o Wolfenstein 3D ou Microsoft Word 6..? :D

Referências:
[1] Windows 3.1 Runs On a Nokia N95, Creating Dangerous Ripple in Space-Time
[2] DOSBox: a x86 emulator with DOS

Echoing hidden psql statements

2009-05-20T10:07:00.001-07:00

One of the greatest advantages of a Database Management System is its embedded data dictionary, also called metadata or system catalog [1]. On PostgreSQL DBMS it is not different, and its metadata are widely used by database handling tools.

A great and yet simple tool available to this relational database is psql [2], a command-line, bash-enabled application that permits issuing SQL instructions to PostgreSQL server instances.

Even when sending a simple command such as "\d" on psql, there might be one or several SQL instructions sent internally to that given instance. Knowing exactly what is being sent could help a database administrator or even a simple developer on common and ordinary daily tasks.

So, how could we know those SQL instructions? Simpler than expected!

There is an environment variable on psql called ECHO_HIDDEN, which once set echoes all hidden instructions, as its own name says. In order to enable that, simply execute the following command:


# \set ECHO_HIDDEN

Then, try to issue some internal commands like "\l" (typing "\?" shows the entire list of options). Here is the output after enabling ECHO_HIDDEN:


# \l
********* QUERY **********
SELECT d.datname as "Name",
       r.rolname as "Owner",
       pg_catalog.pg_encoding_to_char(d.encoding) as "Encoding"
FROM pg_catalog.pg_database d
  JOIN pg_catalog.pg_roles r ON d.datdba = r.oid
ORDER BY 1;
**************************

            List of databases
    Name     |     Owner      | Encoding 
-------------+----------------+----------
 auction5    | sa_auction5    | UTF8
 postgres    | postgres       | UTF8
 rodrigo     | rodrigo        | UTF8
 template0   | postgres       | UTF8
 template1   | postgres       | UTF8
(5 rows)

Note the contents enclosed by "QUERY" and "***". Here are the internal SQL instructions respectively performed.

When you're done, you could deactivate this variable by executing the command below:


# \unset ECHO_HIDDEN

That's it! :D

References:
[1] PostgreSQL 8.3.8 Documentation - System Catalogs
[2] psql - PostgreSQL interactive terminal

Meu PostgreSQL não conecta!

2009-03-25T18:53:00.000-07:00

Começando do início...

Observação: Apesar de os testes terem sido feitos em ambiente Linux, os comandos (ping, telnet, psql) e arquivos de configuração (postgresql.conf, pg_hba.conf) existem e funcionam também no Windows, Macintosh ou FreeBSD, no respectivo terminal.

Antes de tudo, façamos alguns testes que respondem à algumas perguntas.

A máquina está no ar e é enxergada na rede pelo cliente?

Um simples "ping" pode nos ajudar:

$ ping 10.15.23.15

Se estiver tudo correto, aparecerá o texto abaixo:

rodrigo@asgard:~$ ping 10.15.23.15
PING 10.15.23.15 (10.15.23.15) 56(84) bytes of data.
64 bytes from 10.15.23.15: icmp_seq=1 ttl=64 time=0.044 ms
64 bytes from 10.15.23.15: icmp_seq=2 ttl=64 time=0.034 ms
64 bytes from 10.15.23.15: icmp_seq=3 ttl=64 time=0.034 ms

--- 10.15.23.15 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 0.034/0.037/0.044/0.006 ms

Caso contrário, será exibido algo como:

rodrigo@asgard:~$ ping 10.15.23.150
PING 10.15.23.150 (10.15.23.150) 56(84) bytes of data.
From 10.15.23.15 icmp_seq=1 Destination Host Unreachable
From 10.15.23.15 icmp_seq=2 Destination Host Unreachable
From 10.15.23.15 icmp_seq=3 Destination Host Unreachable

--- 10.15.23.150 ping statistics ---
6 packets transmitted, 0 received, +3 errors,
100% packet loss, time 5008ms, pipe 3

Se este for o caso, resolva este problema de conexão e configuração de rede antes de continuar.

O servidor está respondendo ao serviço na porta do PostgreSQL?

Se não estiver, qualquer acesso externo ao PostgreSQL é barrado, e a seguinte mensagem será exibida:

psql: could not connect to server: Conexão recusada
  Is the server running on host "10.15.23.15" and accepting
  TCP/IP connections on port 5432?

Para comprovar isso, podemos fazer um simples teste com o "telnet":

$ telnet 10.15.23.15 5432

rodrigo@asgard:~$ telnet 10.15.23.15 5432
Trying 10.15.23.15...
telnet: Unable to connect to remote host: Connection refused

Este é um problema que ocorre com 5 de cada 4 iniciantes neste banco de dados: a conexão não-local!
Bom, o fato é que a configuração padrão do PostgreSQL faz com que apenas conexões locais (via soquete UNIX) sejam permitidas. O primeiro passo é alterar uma opção no arquivo de configurações postgresql.conf:

Versão	Original	Mudar para
7.X e anteriores	tcpip_socket = false	tcpip_socket = true
8.X em diante	listen_addresses = 'localhost'	listen_addresses = '*'

Após salvar o arquivo, será preciso reiniciar o SGBD (não basta apenas fazer um "reload").

Agora refaça o teste do "telnet". Terá que aceitar a conexão e aparecer este texto:

rodrigo@asgard:~$ telnet 10.15.23.15 5432
Trying 10.15.23.15...
Connected to 10.15.23.15.
Escape character is '^]'.

Dê um CTRL+C para sair. Se ainda não funcionar, será preciso verificar se firewalls não estão impedindo a conexão entre o cliente e o servidor, na porta do PostgreSQL (padrão: 5432). Resolva essa questão antes de continuar a leitura...

Opa! Metade do serviço está concluída! Agora, outro problema que atormenta quem está começando, este erro ao tentar se conectar:

$ psql -h 10.15.23.15 correios rodrigo

psql: FATAL:  nenhuma entrada no pg_hba.conf para máquina "10.15.22.32",
usuário "rodrigo", banco de dados "correios", SSL desabilitado

Sigamos a dica que o PostgreSQL nos dá! Abra o seguinte arquivo de configuração: pg_hba.conf.

Este arquivo controla: quais hosts têm permissão de conexão, como os clientes se autenticam, quais usuários do PostgreSQL podem ser usados e que bancos de dados eles podem acessar. Os registros podem ter uma das seguintes formas:

local DATABASE USER METHOD [OPTION]
host DATABASE USER CIDR-ADDRESS METHOD [OPTION]
hostssl DATABASE USER CIDR-ADDRESS METHOD [OPTION]
hostnossl DATABASE USER CIDR-ADDRESS METHOD [OPTION]
Sendo assim, nestas entradas de permissões de acesso ao PostgreSQL, podemos alterar:

tipo de conexão ("local", "host")
banco de dados ("all": todos)
usuário ("all": todos)
endereço IP e máscara (estilo CIDR)
método ("reject", "trust", "password", "md5", "ident same user")

Importante: o arquivo é lido de cima para baixo, e a primeira entrada que esteja de acordo com a requisição é considerada. Isso é um fato que às vezes confunde os administradores. Os campos podem ser separados por espaços ou tabulações - tanto faz, funcionará de ambas as formas.

Bom, para resolver o problema em questão, precisamos incluir a seguinte linha:

host    correios    rodrigo        10.15.22.32/32        md5

Desta vez, não será preciso reiniciar o PostgreSQL. No Linux, podemos enviar um sinal do tipo HUP das seguintes maneiras:

$ /etc/init.d/postgresql-8.1 reload

$ killall -HUP postmaster

No Windows eu vou ficar devendo, mas deve ser algo como "recarregar o serviço".

Pronto! Simples, não?

Se você quiser, pode fazer com que o PostgreSQL funcione de modo promíscuo, adicionando a seguinte linha:

host    all    all        0.0.0.0/0        trust

Isso faz com que qualquer usuário, de qualquer IP acesse qualquer banco de dados, e sem necessidade de senha!

Se, ao invés de "trust" for usado "reject", todo acesso será fechado.

Lembre-se que, pelo padrão CIDR de endereçamento, diversos IPs podem ser
configurados com uma só linha! Por exemplo, ambas as linhas abaixo fazem
com que toda a subrede do IP 10.15.22.32/22 tenha acesso, por senha, ao
banco "correios":

host    correios    all        10.15.20.0/22        password
host    correios    all        10.15.22.32    255.255.252.0    password

Uma coisa interessante é restringir o acesso ao usuário "postgres", Senhor de Todo o Cluster, com a inclusão da seguinte linha:

local   all         postgres                          ident sameuser

Com isso, somente acesso local (via "ssh" e instrução "su - postgres") poderá ser feita com este super usuário, que tem permissão para fazer o que quiser em qualquer banco de dados.

Setting up Hamachi on Debian GNU/Linux

2009-03-07T18:09:00.000-08:00

"LogMeIn Hamachi [1] is a VPN service that easily sets up in 10 minutes, and enables secure remote access to your business network, anywhere there's an Internet connection."

Hamachi is indeed a great tool for easily setting up VPNs! It creates a virtual network interface and all its configurations are made up with almost no need of user intervention.

For Windows users, a "Next-Next-Finish"-like setup executable is provided. Unix/Linux users, however, have no such facilities and should burn some neurons before putting it to work.

In this article I'll show some tips on how to build up the whole scenario on Debian GNU/Linux operating system, including installing Hamachi binaries, setting up required libraries, and deploying automatic initialization scripts.

First of all, you should download Hamachi binaries for Linux [2]. In the given URL, retrieve the most suitable binary for your processor (i.e. choose between an Intel Pentium or "others"). At the time of writing this very document, the current Hamachi release was 0.9.9.9-20.

For instance, I downloaded hamachi-0.9.9.9-20-lnx.tar.gz as my CPU was an AMD Sempron. I'll use this file name as the example from now on. Save the file on a proper Linux directory, say /usr/src.

Then, get into that destiny directory and extract all the zipped file contents using the following commands:

$ cd /usr/src
$ tar xvzf hamachi-0.9.9.9-20-lnx.tar.gz

Convention: Please observe that in the given notation the prefix "$" means a normal user prompt is needed, whereas a "#" prefix needs a root or superuser terminal to input the commands. Also, texts colored in red are user inputs. Its corresponding outputs are styled in blue.

You should note there is now a sub-directory called hamachi-0.9.9.9-20-lnx. Get into it and then switch to a super user account (i.e. "root" or any supercow-powered user).

$ cd hamachi-0.9.9.9-20-lnx
$ su

Although Hamachi can be set per user, in this case I'll set it up for the entire system, I mean, any user can make use of its networking services, and it will be configured for the "hamachi" account (to be created).

# make install

You should expect some output like this:

Copying hamachi into /usr/bin ..
Creating hamachi-init symlink ..
Compiling tuncfg ..
Copying tuncfg into /sbin ..

Hamachi is installed. See README for what to do next.

Perhaps you get stuck into dependencies, like OpenSSH or OpenSSL. If that is the case, install the required packages (e.g. by calling "apt-get install openssl") before proceeding and then try installing hamachi again. A nice try is to use ldd command onto "hamachi" or "tuncfg" binaries in order to have a clue of what file or package needs to be resolved.

Next, run the TUN/TAP device driver [3] configurator:

# ./tuncfg/tuncfg

This time if nothing comes out everything is alright. :)

The next step consists in generating the user crypto identity:

# hamachi-init

You will see the following output in the terminal:

Initializing Hamachi configuration (/root/.hamachi).
Please wait .. generating 2048-bit RSA keypair .. ok
making /root/.hamachi directory .. ok
saving /root/.hamachi/client.pub .. ok
saving /root/.hamachi/client.pri .. ok
saving /root/.hamachi/state .. ok
Authentication information has been created.

Hamachi can now be started with 'hamachi start' command and then brought online with 'hamachi login'.

Voilà, all the required security keys (private and public) have just been generated on a hidden directory called .hamachi inside the user's home. Note that it is a highly encrypted 2048-bit RSA keypair!

Initialize manually the service by calling the instruction below:

# hamachi start

This single line should appear:

Starting Hamachi hamachi-lnx-0.9.9.9-20 .. ok

Next, it is desirable for you to set a Hamachi nick for the current client by typing:

# hamachi set-nick zangaro

In your case, replace "zangaro" by, say, the identification you usually give your computer.

Now you should put the daemon online and create an account by running this command:

# hamachi login

This simple message is expected:

Logging in ......... ok

If you have no network yet to join, you will need to create yours by typing:

# hamachi create Agajorte

Replace "Agajorte" by the name you will give your new network. A password will be prompted and then the network will be created.

If you are going to join an existing network, just type:

# hamachi join Agajorte

Then, to appear to other users, type this command:

# hamachi go-online Agajorte

Now, to list other members in the network and their respective status, type:

# hamachi list

By default peers' nicknames are not shown in the listing. In order to enable it, you will need to run this command:

# hamachi get-nicks

If you type "hamachi" without any arguments, the outcome is something like this:

Hamachi, a zero-config virtual private networking utility, ver 0.9.9.9-20

version : hamachi-lnx-0.9.9.9-20
pid : 5063
status : logged in
nickname : zangaro

Also, you could have Hamachi usage tips shown by running this command:

# hamachi help

If you successfully came so far, congratulations! We're half way to conclude the process...

You will now need to stop the daemon. Run this command:

# hamachi stop

As every secondary service in the UNIX world, Hamachi daemon could not be initialized using the superuser root account for security reasons. A so-called system user for this service will be created for Hamachi administration tasks. Type the following instruction to add an user called "hamachi" to /etc/passwd.

# adduser --system --disabled-password --no-create-home hamachi

Adding system user `hamachi' (UID 108) ...
Adding new user `hamachi' (UID 108) with group `nogroup' ...
Not creating home directory `/home/hamachi'.
zangaro:/home/rodrigo# id hamachi
uid=108(hamachi) gid=65534(nogroup) grupos=65534(nogroup)

Note that this user has no home directory. He won't need it.

Then, move the Hamachi first initialized configuration directory to /etc/hamachi. In order to let the superuser to still execute Hamachi operations, a symbolic link will be created. Finally, change /etc/hamachi directory and respective ownership to the newly added "hamachi" account. Here are the referenced commands:

# mv /root/.hamachi /etc/hamachi
# ln -s /etc/hamachi /root/.hamachi
# chown hamachi.hamachi /etc/hamachi/ -R

The next step is to develop a Shell Script to start and stop Hamachi daemon. Run this command:

# vi /etc/init.d/hamachi

Then type the instructions below:


#!/bin/bash
#
# hamachi       This shell script takes care of starting and stopping hamachi.
# author: Rodrigo HJORT (http://agajorte.blogspot.com)
#
# chkconfig: 345 99 9
# description: hamachi is a zero-configuration VPN
#

PATH=/sbin:/bin:/usr/bin

HAMUSR=hamachi
HAMDIR=/etc/hamachi
HAMBIN=/usr/bin/hamachi

. /lib/lsb/init-functions

[ -f $HAMDIR/client.pri ] || exit 2
[ -f $HAMDIR/client.pub ] || exit 3

[ -f $HAMBIN ] || exit 4

do_start () {
        echo "Starting hamachi..."
        /sbin/tuncfg
        call_daemon start
}

do_status () {
        call_daemon
}

do_stop () {
        echo "Stopping hamachi..."
        killall tuncfg
        call_daemon stop
}

call_daemon () {
        su $HAMUSR -c "$HAMBIN -c $HAMDIR $1"
}

case "$1" in
  start)
        do_start
        ;;
  stop)
        do_stop
        ;;
  restart)
        do_stop
        sleep 1
        do_start
        ;;
  status)
        do_status
        ;;
  *)
        echo "Usage: hamachi {start|stop|restart|status}" >&2
        exit 1
        ;;
esac

Give proper execution permissions to the script file by running the following command:

# chmod +x /etc/init.d/hamachi

Now you could perform a simple test by calling:

# /etc/init.d/hamachi

Usage: hamachi {start|stop|restart|status}

Then start the daemon:

# /etc/init.d/hamachi start

Starting hamachi...
Starting Hamachi hamachi-lnx-0.9.9.9-20 .. ok

And then retrieve the service status:

# /etc/init.d/hamachi status

Hamachi, a zero-config virtual private networking utility, ver 0.9.9.9-20

version : hamachi-lnx-0.9.9.9-20
pid : 4997
status : logged in
nickname : zangaro-lin

The most important fact to observe now is that Hamachi service is no longer bound to "root" superuser, but to its proper account: the "hamachi" system user.

# ps aux | grep ^hamachi

hamachi 5063 0.1 0.0 3104 804 ? S 01:42 0:00 /usr/bin/hamachi -c /etc/hamachi start

You could also restart the service by invoking this command:

# /etc/init.d/hamachi restart

Stopping hamachi...
Shutting down .. ok
Starting hamachi...
Starting Hamachi hamachi-lnx-0.9.9.9-20 .. ok

At last, you could stop the service by calling this:

# /etc/init.d/hamachi stop

Stopping hamachi...
Shutting down .. ok

Hold on, there is only one detail left to do!

It will be very interesting to have the service initialized automatically as the system starts, i.e. on Linux boot time. This is a responsibility for System-V, but usually its configuration is distribution-dependent. You should check out your Linux distro on how to do it.

As Debian environment was chosen for the tests, System-V style init script links were installed through update-rc.d command, as shown below:

# update-rc.d hamachi defaults 09 99

Adding system startup for /etc/init.d/hamachi ...
/etc/rc0.d/K99hamachi -> ../init.d/hamachi
/etc/rc1.d/K99hamachi -> ../init.d/hamachi
/etc/rc6.d/K99hamachi -> ../init.d/hamachi
/etc/rc2.d/S09hamachi -> ../init.d/hamachi
/etc/rc3.d/S09hamachi -> ../init.d/hamachi
/etc/rc4.d/S09hamachi -> ../init.d/hamachi
/etc/rc5.d/S09hamachi -> ../init.d/hamachi

Well, unless you care about your uptime or if you are sure about the configuration you made on Sytem-V, you might reboot your Linux to find out whether the entire effort was worthy. Bon courage !

Even if you have another kind of Linux, I hope most of information detailed in the present document are valuable for you. Please don't hesitate in scratching up some comments. :D

References:

[1] Hamachi Official Site, https://secure.logmein.com/products/hamachi/vpn.asp
[2] Hamachi for Linux Binaries, http://files.hamachi.cc/linux/
[3] TUN/TAP device driver, http://hamachi.cc/tuntap

Laço FOR em Backup no MS-DOS

2009-02-24T18:42:00.000-08:00

Após um bom tempo acostumado com Shell Script, ter que automatizar processos no Windows (inevitavelmente fazendo uso do MS-DOS...) parece um tanto traumatizante.

Felizmente esse tão esquecido prompt de comando tem sua pseudo linguagem de controle!

Fiz um script de lote, backup-workspace.bat, para realizar o backup de determinados diretórios de trabalho do Eclipse:

@echo off

set ZIP=C:\Arquiv~1\7-Zip\7z.exe

set WKS=%1

echo Backing up workspace %WKS%...

%ZIP% a -r -x@backup-ignore-list.txt -mx=9 %WKS%.zip %WKS%

A tarefa dele é através do 7-ZIP [1] fazer o backup dos arquivos contidos no diretório especificado como argumento no script. Um arquivo com a extensão ZIP e mesmo nome do diretório será criado.

Os padrões especificados neste arquivo de texto, backup-ignore-list.txt, farão com que determinados arquivos e diretórios sejam ignorados:

bin

Thumbs.db

*.log

*.class

org.eclipse.core.resources\.history

org.eclipse.jdt.core

org.eclipse.epp.usagedata.recording

org.eclipse.wst.server.core\tmp0

A última etapa, a criação de um único script para executar os diversos diretórios desejados, precisava ser feita. O novo script, backup-all.bat, poderia ter o seguinte conteúdo:

@echo off

backup-workspace ApressEJB3

backup-workspace Concorde

backup-workspace HiberStruts

backup-workspace PacktEJB3

Achei esse código meio feio, então resolvi dar uma investigada em como deixá-lo menos lusitano...

E eis que encontro o famigerado laço FOR no MS-DOS [2]...!

Agora sim, com umas leves modificações o script ficou assim:

@echo off

set WKS=ApressEJB3, Concorde, HiberStruts, PacktEJB3

for %%w in (%WKS%) do backup-workspace %%w

É, aparentemente ele fez tudo certinho após a execução..!

Referências:

[1] http://www.7-zip.org/

[2] http://home.att.net/~gobruen/progs/dos_batch/dos_batch.html

Artigo na SQL Magazine: "Otimizações com índices reversos"

2009-02-21T17:52:00.000-08:00

"Otimizações com índices reversos", este é o título do artigo que escrevi para a revista SQL Magazine que saiu este mês [1].

Nele descrevo o uso de índices reversos como instrumento para se otimizar consultas SQL em bancos de dados relacionais. No estudo em questão utilizo o SGBD PostgreSQL, comparando o desempenho entre as diversas linguagens procedurais disponíveis na criação de uma função do tipo reverse().

[1] http://www.devmedia.com.br/resumo/default.asp?ed=61&site=2

Le début...

2009-02-21T17:43:00.000-08:00

Voilà, j'ai lancé mon blog enfin !

Cette fois, un enregistrement en français...

EJB 3 Developer Guide: Errata on Web Security Sample

2009-02-21T14:16:00.000-08:00

Today I was just working on the lab 4 / chapter 12 of this book [1] and then I got stuck into an issue...

The environment used on the tests was GlassFish Application Server v2 and the sample code concerns Web Container Security. It just didn't work as expected...

"lab4:

to run program enter http://localhost:8080/BankService/index.html in browser. http://localhost:8080 is url for web applications as shown by Glassfish on startup. If Glassfish provides port other than 8080 then modify url accordingly.

then enter scott/xyz username/password in login form. servlet menu for adding customer to database is then displayed.

enter ramon/abc username/password in login form. error page is then displayed."

The problem is that the page was never displayed despite using the proper user (i.e. "scott") because security always failed.

After googling for a while, I could realize some missing points in the book packaged sample [2].

Yet I made some small modifications in order to test the application with both users: "scott" (belonging to "bankemployee" group) and "ramon" (belonging to "bankcustomer" group).

So, the idea behind it was to forbid access to a specific servlet ("/addCustomer" address) whenever the user did not belong to "bankemployee" role. Also, to restrict access to "/findCustomer" servlet to both roles.

First of all, I modified the WEB descriptor file, "WEB-INF/web.xml", by adding "/findCustomer" servlet mapping, specifying "/addCustomer" URL pattern in the security constraint and an optional welcome file. Here's the entire code:

<?xml version="1.0" encoding="UTF-8"?>
<web-app version="2.5" xmlns="http://java.sun.com/xml/ns/javaee"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://java.sun.com/xml/ns/javaee
http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd">
    <servlet>
        <servlet-name>AddCustomerServlet</servlet-name>
        <servlet-class>ch12lab4.AddCustomerServlet</servlet-class>
    </servlet>
    <servlet>
        <servlet-name>FindCustomerServlet</servlet-name>
        <servlet-class>ch12lab4.FindCustomerServlet</servlet-class>
    </servlet>
    <servlet-mapping>
        <servlet-name>AddCustomerServlet</servlet-name>
        <url-pattern>/addCustomer</url-pattern>
    </servlet-mapping>
    <servlet-mapping>
        <servlet-name>FindCustomerServlet</servlet-name>
        <url-pattern>/findCustomer</url-pattern>
    </servlet-mapping>
    <security-constraint>
        <web-resource-collection>
            <web-resource-name>bank service most restricted</web-resource-name>
            <url-pattern>/addCustomer</url-pattern>
            <url-pattern>/addCustomer.html</url-pattern>
        </web-resource-collection>
        <auth-constraint>
            <role-name>bankemployee</role-name>
        </auth-constraint>
    </security-constraint>
    <security-constraint>
        <web-resource-collection>
            <web-resource-name>bank service restricted</web-resource-name>
            <url-pattern>/findCustomer</url-pattern>
            <url-pattern>/findCustomer.html</url-pattern>
        </web-resource-collection>
        <auth-constraint>
            <role-name>bankcustomer</role-name>
            <role-name>bankemployee</role-name>
        </auth-constraint>
    </security-constraint>
    <security-role>
        <role-name>bankemployee</role-name>
    </security-role>
    <login-config>
        <auth-method>BASIC</auth-method>
        <realm-name>file</realm-name>
    </login-config>
    <welcome-file-list>
        <welcome-file>index.html</welcome-file>
    </welcome-file-list>
</web-app>

The second detail was the creation of a Sun's specific descriptor, "WEB-INF/sun-web.xml", which did not exist in the sample code. The key point here was to map existing groups already specified in the application server's realm file into roles used inside the WEB application. In the current case they share the same names, "bankcustomer" and "bankemployee", on both environments. Here's the new file:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE sun-web-app PUBLIC
    "-//Sun Microsystems, Inc.//DTD Application Server 8.1 Servlet 2.4//EN"
    "http://www.sun.com/software/appserver/dtds/sun-web-app_2_4-1.dtd">
<sun-web-app>
    <context-root>/BankService</context-root>
    <security-role-mapping>
       <role-name>bankcustomer</role-name>
       <group-name>bankcustomer</group-name>
    </security-role-mapping>
    <security-role-mapping>
       <role-name>bankemployee</role-name>
       <group-name>bankemployee</group-name>
    </security-role-mapping>
</sun-web-app>

That's it, it worked at last and I ran the whole book sample codes!

Now I'm gonna take a rest watching "The Hunt for the Red October", which had just finished on the eMule's queue...! :D

References:

[1] Michael Sikora, "EJB 3 Developer Guide", http://www.packtpub.com/developer-guide-for-ejb3/book

[2] Code Download, http://www.packtpub.com/support/book/developer-guide-for-ejb3