Sunday, July 29, 2012

Implement MDM pattern using WSO2 ESB and WSO2 DSS–Part: 1

    In modern complex business data is the key concern out there.Most of the applications try to manage business related data in proper manner with high optimization.One problem comes with data is when organization have multiple data repositories fed with key entities of the business, how to manages that situation without replication key business data. That is where Master data management (MDM) comes in to play.It is an mechanism which can be used to make sure that an organization does not use multiple versions of the same master data in different sections of its operations.In this post I will explain how to achieve one of the patterns in MDM domain using WSO2 ESB (Enterprise Bus) and WSO2 DSS (Data Service Server.) To make the creation and deploying artifacts fast and easy I use WSO2 Developer Studio as the tooling platform.

Imagine a company call AB have key business data in data base and two of its internal department applications want the same key data about employees in different formats. Normal use case is these two departments maintain separate data bases for their own use in preferred format and synchronize those data regularly. With that approach they will have to feel several insolvencies.
  1 . Have to maintain number of data base with same entities.
  2 . Make sure the synchronization happens properly otherwise conflicts will make sure systems not work.
There are others as well but this is not the place to discuss about those.
Below Image will explain the requirement in a proper way. In the image I have shown clearly what are the data formats required by both departments.
How to achieve that?
We can split this task in to two steps.
1. Expose the master data as service which can be consumed by another application.
-WSO2 DSS can expose data in the master data base in the common format.
2.Transform that common service output in to application scope format and consume.
-WSO ESB can transform in to application scope format.
As shown in the below image we can achieve the requirement by using WSO2 ESB and DSS.
I will use WSO2 Developer studio to build the solution you can find installation guide here. We can use Developer studio to create the data service and create two proxy services that do the transformation within the ESB.
Create Sample Data Base.
You can use SQL file here to create the data base and feed it with sample data. I have use mysql the database and the data base looks like below
*Note that in this tutorial I have tried to keep the configuration simple as much as possible and highlight the concept behind that is why I have use simple data base and simple form of data service.
Develop Data Service.
Open the developer studio Dash Board and select Data Service Project shown as below.
Select the option call Create New Data Service in the popup wizard.
Fill the relevant fields as shown below.
In the next wizard page you can configure your data source configurations as shown below.And click on finish.
It will open the data service description language editor where you can edit and configure your data service.
Now we have a configured data source and now we need to add a SQL query to retrieve data from the data base.
Right click on the EmployeeDataService element and select Add Query option from the popup menu.
Give a name for the query and select the data source as mdmdDataSource
Now right click on the newly created query and select Add Sql from the popup menu
As the next step we need to map the data base result to output xml format.To do that right click agin on query element select the Add Outputmapping option from the popup menu.
Fill the Grouped by element and Row name fileds as shown below. Grouped by element will be the root element of the out put xml and row name create employee element for each record in the data table.
To get the out put xml in our own format we need to map the query out put in to our own element this can be done by adding output mapping elements to the result.Click on the employee result element and select add out put mapping category and select element from that.

As shown above our data table has four fields we will map those fields to our output xml elements by filling
select the newly added element and set the Data Source Column Name and OutPut Field Name as shown above image.And likewise we can map other  three fields also.
After that step we need to map above query to web service operation for that data service description language has element call operation.
Once again right click on the EmployeeDataService select Add Operation option from the pop up menu.
Give a name for the operation and since we have a one query in the data service description editor will automatically set the getAllEmployeesQuery for the operation. The final data service will looks like below.

  This data service Exposes the Empolyee data base.
    select * from Employees

Deploy Data Service.

Now its time to deploy and check the result of the exposed data service.

Go the Developer studio dash board again select Carbon Application Project follow the wizard and we will ended up with created Carbon application Project.Open the pom.xml with Carbon Application Project pom editor and select data service project from the dependencies section.

*Note that we have to put mysql-connector.jar to DSS_HOME/repository/components/lib for connection purpose of the DSS and mysql.

Open the standard Servers tab on eclipse right click and select  New –> Server  then go to the WSO2 category select Carbon 3.2 based server as below.


Locate the Data Service server home folder as carbon home folder in the next wizard page.


Click finish and it will add the server to the servers view. Start the server after successfully started the server right click on the dss server and select Add/remove from the menu and add the mdmCar to the server and deploy it. 

Developer studio will automatically open the management console on browser you can log in to it using admin as user name and password go to the web service list on the left hand side menu it will show the EmployeeDataService wich deployed successfully.


Click on the Try this service link relevant to EmployeeDataService it will lead us to try it toll comes with carbon servers it can be used to see the result of out data service.

Click the send button on the left side the result will appear on the right side panel.

19,Quaker Ridge Rd,06801
2 Jeff Firrelli
1000,Coney Island Ave,11230
3 Gerard Bondur
2962,Dunedin CvGermantown,38138
4 Vanauf George
1500,Vance Ave Memphis,38104
5 King Tom
1577,Lobby Ave,38804

That is what we expected !!

In the next part we can learn how to use proxy service to do the transformation fro two department specific formats and complete our MDM solution.Hope to publish second part soon. :)

Please find the car file that contain this completed data service here.