Sunday, July 29, 2012

Implement MDM pattern using WSO2 ESB and WSO2 DSS–Part: 1


Introduction
    In modern complex business data is the key concern out there.Most of the applications try to manage business related data in proper manner with high optimization.One problem comes with data is when organization have multiple data repositories fed with key entities of the business, how to manages that situation without replication key business data. That is where Master data management (MDM) comes in to play.It is an mechanism which can be used to make sure that an organization does not use multiple versions of the same master data in different sections of its operations.In this post I will explain how to achieve one of the patterns in MDM domain using WSO2 ESB (Enterprise Bus) and WSO2 DSS (Data Service Server.) To make the creation and deploying artifacts fast and easy I use WSO2 Developer Studio as the tooling platform.

Scenario
Imagine a company call AB have key business data in data base and two of its internal department applications want the same key data about employees in different formats. Normal use case is these two departments maintain separate data bases for their own use in preferred format and synchronize those data regularly. With that approach they will have to feel several insolvencies.
  1 . Have to maintain number of data base with same entities.
  2 . Make sure the synchronization happens properly otherwise conflicts will make sure systems not work.
There are others as well but this is not the place to discuss about those.
Below Image will explain the requirement in a proper way. In the image I have shown clearly what are the data formats required by both departments.
Scenario
How to achieve that?
We can split this task in to two steps.
1. Expose the master data as service which can be consumed by another application.
-WSO2 DSS can expose data in the master data base in the common format.
2.Transform that common service output in to application scope format and consume.
-WSO ESB can transform in to application scope format.
As shown in the below image we can achieve the requirement by using WSO2 ESB and DSS.
          image
I will use WSO2 Developer studio to build the solution you can find installation guide here. We can use Developer studio to create the data service and create two proxy services that do the transformation within the ESB.
Create Sample Data Base.
You can use SQL file here to create the data base and feed it with sample data. I have use mysql the database and the data base looks like below
image
*Note that in this tutorial I have tried to keep the configuration simple as much as possible and highlight the concept behind that is why I have use simple data base and simple form of data service.
Develop Data Service.
Open the developer studio Dash Board and select Data Service Project shown as below.
dsp1
Select the option call Create New Data Service in the popup wizard.
dsp2 
Fill the relevant fields as shown below.
image
In the next wizard page you can configure your data source configurations as shown below.And click on finish.
image
It will open the data service description language editor where you can edit and configure your data service.
image
Now we have a configured data source and now we need to add a SQL query to retrieve data from the data base.
Right click on the EmployeeDataService element and select Add Query option from the popup menu.
image
Give a name for the query and select the data source as mdmdDataSource
image
Now right click on the newly created query and select Add Sql from the popup menu
image
As the next step we need to map the data base result to output xml format.To do that right click agin on query element select the Add Outputmapping option from the popup menu.
image
Fill the Grouped by element and Row name fileds as shown below. Grouped by element will be the root element of the out put xml and row name create employee element for each record in the data table.
image
To get the out put xml in our own format we need to map the query out put in to our own element this can be done by adding output mapping elements to the result.Click on the employee result element and select add out put mapping category and select element from that.
image

As shown above our data table has four fields we will map those fields to our output xml elements by filling
         image
select the newly added element and set the Data Source Column Name and OutPut Field Name as shown above image.And likewise we can map other  three fields also.
After that step we need to map above query to web service operation for that data service description language has element call operation.
Once again right click on the EmployeeDataService select Add Operation option from the pop up menu.
image
Give a name for the operation and since we have a one query in the data service description editor will automatically set the getAllEmployeesQuery for the operation. The final data service will looks like below.


  This data service Exposes the Empolyee data base.
  
    root
    abc123
    jdbc:mysql://localhost:3306/abcompany
    com.mysql.jdbc.Driver
  
  
    select * from Employees
    
      
      
      
      
    
  
  
    
  


Deploy Data Service.

Now its time to deploy and check the result of the exposed data service.

Go the Developer studio dash board again select Carbon Application Project follow the wizard and we will ended up with created Carbon application Project.Open the pom.xml with Carbon Application Project pom editor and select data service project from the dependencies section.

*Note that we have to put mysql-connector.jar to DSS_HOME/repository/components/lib for connection purpose of the DSS and mysql.

Open the standard Servers tab on eclipse right click and select  New –> Server  then go to the WSO2 category select Carbon 3.2 based server as below.


image

Locate the Data Service server home folder as carbon home folder in the next wizard page.


image

Click finish and it will add the server to the servers view. Start the server after successfully started the server right click on the dss server and select Add/remove from the menu and add the mdmCar to the server and deploy it. 

Developer studio will automatically open the management console on browser you can log in to it using admin as user name and password go to the web service list on the left hand side menu it will show the EmployeeDataService wich deployed successfully.


image

Click on the Try this service link relevant to EmployeeDataService it will lead us to try it toll comes with carbon servers it can be used to see the result of out data service.

Click the send button on the left side the result will appear on the right side panel.


   
      1
      Diane
      Murphy
      
19,Quaker Ridge Rd,06801
2 Jeff Firrelli
1000,Coney Island Ave,11230
3 Gerard Bondur
2962,Dunedin CvGermantown,38138
4 Vanauf George
1500,Vance Ave Memphis,38104
5 King Tom
1577,Lobby Ave,38804



That is what we expected !!

In the next part we can learn how to use proxy service to do the transformation fro two department specific formats and complete our MDM solution.Hope to publish second part soon. :)

Please find the car file that contain this completed data service here. 

8 comments:

  1. This comment has been removed by the author.

    ReplyDelete
  2. It's helpful. Thank you.
    I am very need the scenario of "how to use proxy service to do the transformation fro two department specific formats and complete our MDM solution".

    ReplyDelete
    Replies
    1. 來自大海的心 , Sure I have done most of the 2nd part and I will publish it within this week thanks.:)

      Delete
  3. This comment has been removed by the author.

    ReplyDelete
  4. Hi Kalpads,

    I have one Query
    Here i am sending the my GIT URL which contain my Query

    https://gist.github.com/anonymous/5300686

    Could you please help me.
    Thanks in Advance
    Anil

    ReplyDelete
    Replies
    1. Hi Anil,

      your question was :- How can i create in Esb proxy service.

      Please find the proxy service creation flow using WSO2 Developer Studio at [1]. And also you can find other related
      information about deployment also in that documentation. And if you have faced any other issues please ask on Stack Overflow our
      guys keep eye on SO you will get the answers promptly.

      [1]--http://dist.wso2.org/products/developer-studio/2.1.0/docs/esb_category.html#ProxyServiceArtifact

      Thanks
      Kalpa

      Delete
  5. hi kalpa, i m unble to create wso2 carbon 3.2 based server plz help me regarding this .. thank u

    ReplyDelete
  6. Hi... When i start with an ESB project and create the Endpoints, Sequences and proxy services then try to do a Composite Application it will generated poml.xml file. But when i try to create the Archive(.car) it will not be generated. Can anyone please help me out with this? It would be gratfull.

    ReplyDelete