I came across an open source, low cost, EAI tool the other day. Jitterbit . I’ve talked about the need for a low-cost EAI tool before, and was hoping that Sun would say something at this years Seebeyond conference, but nothing.
I took jitterbit for a spin. It’s not bad. It’s not as fancy as other 6 figure tools, but that’s ok. It stores every definition in text (xml) files, which I like. It handles mildly complicated text files and jdbc integration – which from my experience accounts for 60% of all integration needs. It also seems to do decent web service integration.
It has this notion of jitterpacks which are prebuilt integrations that can be shared. I’m unsure how useful such a thing might be, as semantics may get in the way. In any case, jitterpacks could be good.
Currently, it doesn’t integrate into any JBI, or messaging platforms such as servicemix or plain jms, but I’m sure with time this will come.
The forums suggest that there will be a workflow (composition) component coming out, which is most definitely needed. All in all, looks very promising.
The last thing I like is that it isn't a plugin to eclipse. I find eclipse ugly and unweildy. I must be the only one. I didn't bother looking to see if its eclipse underneath, but if it is - good on them.
[update: and it seems to be getting some press too]
Very interesting series on what happened at a conference called “BPM Think Tank”. Lots of interesting advanced thinking on BPEL, XPDL, BPMN, BPDM and what it all means, if anything at all.
The conference is run by the OMG, and is the place where last year BPMN and Activity diagrams were “confluenced”.
The main take away for me was the facts around fidelity of transformation between different models and the fact that BPEL may not have legs in it. I recommend the whole series of those interested in BPM topics.
Actually, there was one more thing, a new standard called Business Process Runtime Interface, which is a common way of getting process metrics out of tools. Not sure what it'll mean, but it sounds like it'll be better than a) building your own process runtime monitoring tool, b) writing code to get it out of proprietary repositories.
Disambiguity tagged a product called Lucid Spec. Lucid Spec if a Gui prototyping/requirements tool. If you've ever been involved in requirements gathering you'll know that quality tends to zero. One way I've been trying to stop this is by drawing screen mockups using visio - with double-click navigation behaviour too!
I've been keeping my eye on a couple of other tools in this space, axure and irise (whose website seems to be down, or they've gone out of business), waiting for a project where we can test these kinds of tools out. These things are a worthwhile addition to a BA's toolkit.
I know from experience, that wireframes help with clarity, yet it's amazing how many specs try and use words to describe things, all in the name of not doing "implementation" during requirements.. [sigh]..
David Nicolette writes about John Kern presenting at his company about Agile stuff. It really highlights the disconnect between different levels of thinking on a topic. If you're thinking has evolved significantly in a topic, it's hard to talk back at an unevolved level.
I've had a bit of experience at this, and now I try and get into my audience a bit more. Being a change agent is really fricken hard. I've almost come to the conclusion that the only thing that matters from an audience's perspective is, "what's in it for me?". If I can answer that, I think change may be possible. If I can't, I'd be wasting my time. It doesn't always stop me though.
I don't really have much to add about the article, it sits as an excellent slice on the difficulty of changing a group of people.
I came across this interesting article about the intersection of collaborative tagging and top-down taxonomies. It brings to light the idea of Pace Layering, which is the idea that different layers of things have different change rates, eg, housing, software and information tagging. It applies this concept as well as a couple more to knowledge taxonomy creation within enterprises.
Awfully interesting if you’re trying to figure out how to bring bottom-up tagging into your knowledge management environment.
Software developers are the clerks of the 21st century. What do I mean? I mean software development will become a workflow execution problem, just like any other clerical task. Everyday, you will look at your in-tray and work through the items there. When you’re finished with the work item, you’ll stamp it “done” and put it in your out-tray.
Why do I say this? Mostly, because I’ve worked in an environment almost like this. It was close. And it’s coming closer. Here’s some trends that all lead me to this conclusion.
Compliance initiatives for both legal and Enterprise Architecture reasons will make middle management wish it could see what was going on. You’re WBS will be assigned and managed by a workflow system. Deviance from estimate will be tracked. This data will feed back into the IT Governance tooling thus allowing a better understanding what’s happening on the shop floor.
Microsoft is entering this market in a big way. I can’t figure out the pricing on this, but it could(should) be cheap. Certainly much cheaper than the cost-prohibitive tools of Rational a few years ago.
Microsoft isn’t the only vendor in this space. There are quite of few players trying to provide Application Lifecycle Management. Vendors such as Borland, IBM(nee Rational), Compuware, Telelogic, plus more. They’re all out there, beating this drum, selling the vision to managers. Which PMI trained manager can resist – you can’t improve what you can’t measure!
In the open source world, there’s a subtle happening too. Jira is available free for any open source projects as an "issue tracker", and it’s used by many. Unbeknownst to many people is that jira is really a generalised workflow management system that just happens to have an instantiation as a bug tracking tool. I’ve seen on the mailing lists that people are using it for all sorts of development artefacts, as well as for non-development workflows. What this is doing is socialising people to the idea of tracking everything in a tool.
More generally, the world is becoming socialised to BPM tooling in many different fields. BPM(nee workflow) is being used to support many organisational processes, helpdesk, ITIL, core business. The maturing of BPM thinking will mean people start seeing software development as a BPM problem.
What would a post about software development be without a mention of Outsourcing? Outsourcing’s requirement to increase visibility is also putting pressure in this area. Selling visibility to a CIO is gold. You can use phrases such as "Process Transparency" with wild abandon and they lap it up.
Now that I look back on this post, I’m thinking I can use some of its ideas as manure in my own company to help seed the implementation of a generalised work item system. Of course, the trick, as it always is, is to align these systems with our humanity.
Anyway, I hope you get this memo. As always comments welcome.
I’ve finished reading “The Goal”. I’d just like to say it’s an excellent read. I’ve read a fair amount of business books in my time and this novel approach is quite clever. Makes it seem like reading for pleasure instead of reading for work.
I read this book because I found it referred to in Agile Management for Software engineering. Agile Management is an attempt to apply the techniques described in The Goal to software world. If you do read Agile Management, or have read it, do read The Goal.
All those software types with methodological bents should read The Goal. It really makes you think about clarifying why you’re building software.
I’ve spent some time looking into IBM’s Common Base Event ideas. The driver for my interest comes from my experience in enabling auditability/operability of integration solutions. I’ve seen a number of logging frameworks for event tracking and they were all inconsistent with the type of data they captured. So I was looking for some “canonical” event model.
Common Base Events are just such a thing. Basically, it’s an XML document that defines an event at either the technical or business level. I’d like to break this discussion up into two different sections, technical events and business events.
Technical Events
Why would you want to use CBE and not just use log4j or something?
As we all know, interpreting different system’s log files requires considerable experience with the system. Sometimes, the log file format is defined, other times it’s up to the competence of the system operators to guess what it means. CBE is an attempt to standardise and unify events and situations so that operators have a better chance of resolving a situation. This standardisation doesn’t only help operators, it allows for “self-healing” of an application, as management applications can more easily ascertain problems and therefore solutions.
The team that defined this canonical studied many different logging styles and came up with the model specified. It’s apparently very similar to some work done by the Tivoli group.
In many ways, this canonical event model allows easier problem management. This event model is also part of the WSDM initiative.
Business Events
This is the thing I was really interested in. I want to log when a SalesOrder hits my system, I then want to log the transform&enrichment I do to the canonicalSalesOrder. Lastly I want to log its change to a SAP iDoc and then I wish to log the response I get from SAP. In effect, I wish to log all state changes. This log, I wish to expose to Line Of Business users rather than technical folks. There’s a big section on business events in this best practices document. IBM is using this framework to power their BAM solution. The IBM Business Modeller allows the definition of an Event Model in the Business Process, which is quite like Oracle’s Sensor concept. So, it seems like I can do what I want, though it feels a little like I’m shoehorning a techo solution for a business solution.
Where things can be improved
It would be great if I could download a reference implementation of the CBE unencumbered by IBM products, in particular Websphere and eclipse. This reference implementation would contain all the pieces for me to implement this in my application, including the Common Event Infrastructure and a repository. It would also be great if a simple webapp was also included that served as a front end, as a sort of sample viewer. The reason why this clean-room RI is needed is that it will be easier to convince application stakeholders of the goodness of this approach if it’s all simple, easy and mature. A .Net sample client implementation would also be quite handy to convince those VB programmers that logging is cooler than it sounds.
CBE’s relationship to Non Functional Requirements
CBE affects two Quality Attributes. Operability is affected by the design of the event system. If all components of an application, ie, database, web-server, app-server etc, all emit CBE’s then the ability of operations staff to manage problems is improved. Auditability can also be enhanced if the correct business events are emitted.
CBE’s and their standardisation in the application allow us to start measuring conformance to design. For example, if we design a CBE event model, it will be possible to scan the code base for conformance to this model. Additional to static scanning, it will be possible to append test cases with checks for conformance to expected event emission. This last paragraph on operationalising Quality Attributes is a nascent idea I’ve been thinking through. Expect a more thorough treatment in the near future.
CBE and Legacy
Obviously, you cannot rewrite all your logging code to meet this new format. IBM has provided us with a couple of solutions. Firstly, they’ve built a log file adapter. A tool that parses log files and emits CBE events. It’s a bit like an EIS adapter for log files if you will. The other slightly more invasive method is to slip in a CBE implementation underneath the Commons Logging, or Log4j or java.util.logging frameworks.
CBE Summary
All in all these technologies help put some concrete meat around the concepts of auditability and operability – surely a step in the right direction.
Over the last couple of weeks I’ve become concerned with the
tone of this writing. I find it easy to fall into a shrill criticism of things.
This criticism does nothing for nobody and I get there simply because it’s
easier to point and ridicule rather than discuss and suggest. I will endeavour to
frame everything within a more constructive tone from now on.
eGate integrator is the core of the platform. It defines the foundational abstractions that drive the platform. The record structure that is manipulated is called an “Object Type Definition” (OTD). OTDs are created from source types and can represent any other record type, eg XML, BAPI, SEFs, Flat File, Relational row, WSDL, etc. You can think of it as the canonical type definition used by the platform. The OTD is a java class. It does all the marshalling/unmarshalling to these various formats.
The Connectivity Map is a bit like a service orchestration layer. It strings together the adapters with the transformation maps with the services. It’s a simple straight through processing pipeline.
Collaboration Definitions are the transformation component of the tool. There are two types, java and xslt. They call these things Business Rules. Which is a bit of a misnomer, don’t you think? What the Java “Business Rule Editor” does is pretty funny. It is a point&click interface sitting over the top of java. Yep. That’s the extent of this transformation language. So in essence the “Business Rule Developer” needs to have a thorough grasp of java libraries and debugging. The only thing that’s been abstracted is syntax. Since only a java developer should really do one of these mappings, do you think they want to create variables, or fields, or try/catch blocks by point&click? I noticed that there is no filtering of the jdk, meaning I can create jFrames on the server. This transformation language reminds me of the Active Software tool, now this, that had a somewhat similar paradigm. It was clever, for 1998. I guess the fundamental issue I have with this approach is that it doesn't raise the abstraction level. I'm still at the java level when really I should have some kind of DSL for this very constrained domain.
The other language of transformation is XSLT. Can’t say I’m a fan of this either. XSLT quickly becomes write-once software. As in, you write it once and hope it never needs changing, because it’s too difficult to read, understand or god forbid, change.
I do hope we see Sun introduce an Xquery transformer. Xquery is a lot easier for regular joes like me to get my head around. BEA has a nifty xquery mapper which you can read about here.
EIS Adapters are called eWays. Look like every other adapter suite.
From a deployment perspective, it looks pretty fancy. Looks like you can deploy collaborations to all sorts of logical hosts. What I mean by this is that you can get services running in a distributed manner on many different machines. This makes sense when you consider the history of this product, being fundamentally a message broker. Most interestingly, it can deploy into common runtimes such as weblogic or websphere. This is pretty amazing really and makes the TCO so much more palatable for deployments. Looking through some of their generated artefacts, it looks like they’ve really tried hard to make SeeBeyond fit within the j2ee platform.
The other thing I’m a bit concerned about is the “Repository”. The repository stores all the artifacts. I’m not a big fan of repositories. It usually means I can’t use my standard build or config management processes. Can’t use my standard file compare tools. Can’t see the current state of the branch… not having this stuff really hurts you in the few weeks before and after a production deployment. I’m not a fan of repositories.
In terms of testability, well, I don’t know about this either. I can’t see how I can do any unit testing of any parts of the tool. They provide a manual testing tool for the mapping artefact, but it doesn’t look like I can hook that up to a build environment and execute them in the “automated test” kind of way. I guess if I was clever and more experienced in the platform I could figure out how to intercept the JMS calls and do asserts on outputs from the queues or something. Looks like I’ll need deeper experience before I can be sure. But one thing is for sure, they don’t make it easy to know how to setup a testing strategy.
The documentation I’m working from is a little light. It seems aimed at beginners and intermediate folks. There doesn’t seem to be documentation at the advanced level. Things such as api docs, or other deep sides of the tool. Of course, there may be this info on the customer site.. but who could know?
Just noticed that the Designer tool is based on netbeans. How lucky for Sun.
