Jul 122012
 

I recently changed my job and so I have now new duties and responsibilities. Mainly understanding devops processes as they are currently and then scoping out improvements, introducing new tools, processes etc. I’m a master on drafting everything on whiteboards, however there are 2 drawbacks:

  • nobody except me seems to be able to read my handwriting, so sending out photos of whiteboards only causes WTF’s
  • it’s kinda hard to annotate, rearrange or optimize what has been written down and photographed

So I had a look how to easily translate my drawings into something readable. I don’t own a visio license, I checked a few commercial tools but didn’t find the one that was made for me in terms of simplicity, usability and flexibility. 

My first find was yEd, a nice editor to easily draw diagrams. I did do a couple of server/network layouts with it, documented processes and it worked out quite well. But then there was a graph i couldn’t draw – basically a version control branching strategy for a set of enterprise products. I picked up one of my co-workers presentations and followed up with drawing in powerpoint which was better than expected but not quite satisfying. 

Today I had to draw just another graph and neither yEd nor Powerpoint fitted the requirements:

  • I wanted to easily extend/modify the objects
  • I did not want to arrange object manually
  • I didn’t want to get distracted by moving pixels but i wanted to focus on content. 

I decided to use dot language as it is plain text, generates graphs and takes care about the alignment of objects itself. I used dot indirectly a few days ago when I used vizant to generate a documentation over a set of ant files and jaranalyzer to outline dependencies between jar files. The dot documentation looked a bit to brief for me so I did use the sample gallery and just picked a graph that contained something I needed and 2h later I ended up with my complete whiteboard in a nice and presentable way. All change request I implemented within minutes and the output always was a nice looking, technical graph. I used graphviz for visualization and export to image files. Graphviz is quite sweet, it loads file changes straight away, so I had my text editor and graphviz side by side to get an instant preview.It seems there is a reasonable community using it as all my questions were asked before and got answered at stackoverflow.com.

Here a little example for a graph:

digraph “G” {

  // general config

node [style=filled, color=grey97, fontname=Helvetica, fontsize = 12];

edge [color=grey50]

// ranking

{rank = same; start1, “start2″}

{rank = same; aDone, bDone}

// boxes

    subgraph cluster_a {

    [ label="a box", color=lightsteelblue1, style=filled;fontcolor=dodgerblue3];

subgraph cluster_aa {

    [ label="a box in a box", color=lightsteelblue2, style=filled;fontcolor=dodgerblue3];

    organge ; normal;

   

longLabel; 

}

 

   subgraph cluster_b {

   [ label="another box", color=gray90, style=filled; fontcolor=dodgerblue3];

taskB

      -> bDone;

    } 

 

    subgraph cluster_c {

   [ label="invisible box", color=white, fontcolor=dodgerblue3];

anotherLabel; 

}

 

    start1 -> taskA;

 

    “start2″ -> {normal organge } -> join -> taskA -> aDone  -> taskC -> done;

    “start2″ ->  longLabel -> anotherLabel -> join ;

 

    join -> taskB [label="edge label", fontcolor=grey];

    // node aliases and configuration

    node aDone [label="a",color=green2]

    node bDone [label="b",color=green4]

    node longLabel [label="a long label\nwith line break"]

    node join [label="join and fork"]

    node anotherLabel [label="another label"]

    node organge [label="organge",color=bisque2]

node done [color=grey80]

}

Below is the generated image:
  • it contains layout configurations for nodes and edges (colors, fonts)
  • boxes of type cluster
  • color highlightings
  • edge lables
  • ranks to aling elements

this is a subset of what I have done today. Considering that I looked into it today for the first time and that i got everything the way I wanted I’m quite happy. I admit a graph like this can be easily done in any other visualization tool. However, the power of dot is in how simple it is to do modifications. Anohter sub graph or another order of elements won’t require major reconstruction work. simply change the order and that’s it. 

graph output

Popularity: 19% [?]

Tags: , , , , , ,
Jun 222011
 

Due to a recent project I started working on a talend component. This blog post is not about how to do this from scratch, I am quite sure that Talend provides enough tutorials about it, alternatively you can do it like me and just use existing components as reference and build your own one (work environments without internet access suck!)

Prior reading this You Should Have:

  • A basic understanding of Talend Data Integration solutions. 
  • A basic understanding how to develop and deploy Talend (Java) components.
  • Knowledge about maven, maven build plugins and ant builds.

My Environment is:

  • Mac OS 10.6.6 with Java 1.6
  • Eclipse 3.5.1 Java EE editon
  • Jaspersoft ETL 4.1.2 (aka Talend Open Studio 4.1.2)

 

Everything I describe applies for Jaspersoft ETL, Talend Open Studio and Talend Integration suite. The <Jaspersoft ETL workspace dir> is my local Talend repository folder – the Eclipse workspace folder of Jaspersoft ETL.

 

At The Beginning I found a few items to struggle with:

  • Project structure: 
    • src/main/template: contains all files which are patched during the build process such as the component registration file.
    • src/main/resources: contains all resources such as jet templates, icons, message properties
    • src/test/java: reference implementation of the functionality which shall be in the component
    • for the build (see below):
      • src/assembly, pom.xml for the maven build
      • build.xml for the ant wrapper.
  • Editing Template files: By default Eclipse handles *.javajet files as text files without code completion and syntax highlighting. If you download the m2t JET from http://download.eclipse.org/modeling/m2t/updates/releases/ an editor will be installed. Now go to Preferences > General > Editors > File Associations and add “*.javajet” as extension. Select it and click on the second Add.. button and select “JET-Java Emitter Templates Editor”. Click Ok. Reopen the javajet templates. 
  • To refresh the javajet templates in Talend without restarting it just hit Control+Shift+Function+F3 to re-initialize the code generation.
  • Dealing with JETorg.eclipse.emf.codegen.jet.JETException: The generated java files are in a specific folder. These files can help to understand what the compilation error caused. I linked the folder into my project
    ln -s <Jaspersoft ETL workspace dir>/.JETEmitters/src/org/talend/designer/codegen/translators/<family> ./gen-src
    ln -s <Jaspersoft ETL workspace dir>/.metadata/.log ./log.log
    The log file i monitor with tail on command line.  

The Build And The Component File

I setup maven to maintain the dependencies for my project as I don’t want to collect everything manually, I also want to be able to easily migrate to a later version. Now, as you might now, the component descriptor file requires a list of component specific imports. For obvious reasons I did not want to create this list manually, therefore I wrote a little ant script which wraps the maven build and patches the component descriptor.

The maven build file is responsible to collect all required files and bundle it as a single zip file. I used

  • the maven dependencies plugin to download all required jars into the target folder during prepare-package phase.
  • the maven assembly plugin for collecting all resources and bundling into a zip file during package phase.

The ant build is has only a few targets

  • a prepare target which triggers mvn clean prepare-package
  • a patch method which utilizes the (evil) antcontrib library to iterate over the jars and collect their names which are written in the required structure into the component file. 
  • a package target which triggers mvn package.

 

Popularity: 33% [?]

Tags: , , , , , , , ,