How To Develop And Test Custom Xpath Function For Xalan.

Introduction

  XSLT 2.0 offers a rich set of transformation tools for xml transformation. But still in some cases we may encounter very specific business data transformations which can not be handled by generic transformation function. To handle those cases we may have to use the extension capability of the underling processing engine.

Problem

 Lets imagine a legacy backend system responding with following xml payload.

 It follows the upper camel case format to construct the xml payload.

	<?xml version="1.0" encoding="UTF-8"?>
	<Customers>
	<Customer>
	<Name>Kalpa</Name>
	<CardNum>786543-8739-6373899</CardNum>
	</Customer>
	</Customers>

view raw input hosted with ❤ by GitHub

 And the mobile client which consumes the system payload expects lower camel case format to process the payload.

	<?xml version="1.0" encoding="UTF-8"?>
	<customers>
	<customer>
	<name>Kalpa</name>
	<cardNum>786543-8739-6373899</cardNum>
	</customer>
	</customers>

view raw out hosted with ❤ by GitHub

Solution

  We are going to implement extension function using Xalan XSLT processor. Let's divide the solution in to few steps.

   1.Create the Java implementation of the logic.

       Google guvava api has a rich set of string manipulation tools. We are going to reuse one of those for converting UpperCamelCase string to lowerCamelCase string.

	package custom.xpath;
	import com.google.common.base.CaseFormat;
	public class CustomXpath {
	/**
	* Returns lower camel case of the given String input.
	*
	* @param elementName local name of the xml element.
	* @return Converted String as lower camel case.
	* @see CaseFormat
	*/
	public static String toLowerCamelCase(String elementName) {
	return CaseFormat.UPPER_CAMEL.to(CaseFormat.LOWER_CAMEL, elementName);
	}
	}

view raw CustomXpath hosted with ❤ by GitHub

   2. Plug the custom function to XSLT.

       
       We have define new namespace with following pattern.

       xmlns: prefix ="xalan:// packagename.classname "

       In our example we have 

          xmlns:custom = "xalan://custom.xpath.CustomXpath"

       And we can call the function as follows

         {custom:toLowerCamelCase(local-name())}

	<?xml version="1.0" encoding="UTF-8"?>
	<xsl:stylesheet version="1.0"
	xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:i="http://www.w3.org/2001/XMLSchema-instance"
	xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:custom="xalan://custom.xpath.CustomXpath">
	<xsl:output method="xml" version="1.0" encoding="UTF-8"
	indent="yes" />
	<xsl:template match="node()"/>
	<xsl:element name="{custom:toLowerCamelCase(local-name())}">
	<xsl:value-of select="text()"></xsl:value-of>
	<xsl:apply-templates select="node()|@*" />
	</xsl:element>
	</xsl:template>
	<xsl:template match="text() | @*">
	</xsl:template>
	<xsl:template match="@*">
	<xsl:copy>
	<xsl:apply-templates select="node()|@*" />
	</xsl:copy>
	</xsl:template>
	</xsl:stylesheet>

view raw customXpath hosted with ❤ by GitHub

    3. Test the custom Xpath function.

       As the final step we are creating a Junit test for test the custom Xpath function. One thing to note here is we are not using the default xslt processor instead we set the transformerfactory as "org.apache.xalan.processor.TransformerFactoryImpl" so that transformation will be done using Xalan apis.

	package custom.xpath;
	import java.io.File;
	import java.io.FileNotFoundException;
	import javax.xml.transform.Transformer;
	import javax.xml.transform.TransformerException;
	import javax.xml.transform.TransformerFactory;
	import javax.xml.transform.stream.StreamResult;
	import javax.xml.transform.stream.StreamSource;
	import junit.framework.TestCase;
	import org.junit.Test;
	public class CustomXpathTest extends TestCase {
	public CustomXpathTest() {
	}
	@Test
	public void testToLowerCase() throws FileNotFoundException,
	TransformerException {
	System.setProperty("javax.xml.transform.TransformerFactory",
	"org.apache.xalan.processor.TransformerFactoryImpl");
	transform("src/test/resources/input.xml",
	"src/test/resources/customXpath.xslt",
	"src/test/resources/out.xml");
	}
	public static void transform(String sourcePath, String xsltPath,
	String resultDir) {
	TransformerFactory tFactory = TransformerFactory.newInstance();
	try {
	Transformer transformer = tFactory.newTransformer(new StreamSource(
	new File(xsltPath)));
	transformer.transform(new StreamSource(new File(sourcePath)),
	new StreamResult(new File(resultDir)));
	} catch (Exception e) {
	e.printStackTrace();
	}
	}
	}

view raw CustomXpathTest hosted with ❤ by GitHub

Conclusion
Developing a custom Xpath function and plugin it to your code is not that hard but always look for existing Xpath function or existing extension function set like EXSLT. After all its good old Java , open for you to play with.  

You can find the eclipse project with the sample code at  https://github.com/KalpaD/xalan_custom_xpath_sample

Mocking java.net.URL

Introduction.

Mocking is a crucial part of the unit testing. It gives power to the developer to push the limits of the logic without pleasing the dependencies of the logic. Mocking has it's own forms for different testing paradigms and also limitations. Recently I have used mockito mock library to develop some unit test cases. Even though Mokito is grate tool for mocking it has few limitations ,by design Mokito can not mock/spy final classes, anonymous classes or primitive types. Here I will explain how I have found a workaround for final class problem.

Problem
Have a look of "public String getContent(URL url) throws IOException" method which takes java.net.URL as an argument and read the input stream from the http connection. Here I need to mock java.net.URL is a final class we often see in our codes , because of above limitation we discussed above I can not get HttpURLConnection form mocked URL instance.

     public class WebContentManager {

    private final String USER_AGENT = "Mozilla/5.0";
   
    public String getContent(URL url) throws IOException {
   
     HttpURLConnection httpConnection = null;
     BufferedReader in = null;
     StringBuffer response = new StringBuffer();
   
     try {
      httpConnection = (HttpURLConnection) url.openConnection();
      httpConnection.setRequestMethod("GET");
      httpConnection.setRequestProperty("User-Agent", USER_AGENT);
   
      // int responseCode = httpConnection.getResponseCode();
   
      in = new BufferedReader(new InputStreamReader(
        httpConnection.getInputStream()));
      String inputLine;
   
      while ((inputLine = in.readLine()) != null) {
       response.append(inputLine);
      }
     } finally {
   
      try {
       in.close();
      } catch (IOException ex) {
   
      }
      if (httpConnection != null) {
       httpConnection.disconnect();
      }
     }
   
     return response.toString();
    }
    
   }
   

Solution
So I started to dig in to other ways to create mock URL instance with injected mock HttpURLConnection. java.net.URL class has constructor which takes java.net.URLStreamHandler as an argument. java.net.URLStreamHandler is an interface with the method  protected URLConnection openConnection(URL u ) throws IOException.So if I can inject the mocked HttpURLConnection in to URLStreamHandler openConnection() method.
Here goes the unit test implementation.

   public class WebContentManagerTest {

    String response;
    
    @Before
    public void setUp(){
     response = "";
    }
    
    @Test
    public void testGetContent() throws IOException {
     
          final HttpURLConnection mockCon = mock(HttpURLConnection. class);
          InputStream inputStrm = IOUtils.toInputStream( response);       
          when(mockCon.getLastModified()).thenReturn((Long)10L, (Long)11L);
          when(mockCon.getInputStream()).thenReturn( inputStrm);
          
           //mocking httpconnection by URLStreamHandler since we can not mock URL class.
          URLStreamHandler stubURLStreamHandler = new URLStreamHandler() {
              @Override
               protected URLConnection openConnection(URL u ) throws IOException {
                  return mockCon ;
               }           
          };
          
          WebContentManager wcm = new WebContentManager();
       URL url = new URL(null,"https://www.google.com",stubURLStreamHandler);
          String actual = wcm.getContent(url);
          assertEquals(response, actual);
   
    }

    }
   

Conclusion
When you find final class that can not mock , try to find alternative ways to create the object and inject what you wanted in to that. 




    

Enable two factor authentication for your web application to use with Google Authenticator App.

Introduction.

If you have already enabled “2-Step Verification” for your Google account (Or any other)you may aware of what we are going to discuss today. For all others “2-Step Verification” is  simply another mechanism to keep safe your Google account from the “bad guys”.

Basically what happens is when you enable the 2-Step Verification (more generically two factor authentication) addition to the your user name password combination it will ask for a verification code as the second gateway to your account. As the primary option Google offers SMS or voice call as the delivery mechanism for the verification code.

Or we can switch to “Google Authenticator App” which is available freely on Play Store and App Store. In this post I am going to discuss how we can implement our own 2-Step Verification for Google Authenticator App is the client side of this story, and google has their own server side to enable service for google users and we are going to have our own today.

This mechanism powered by TOTP- Time Based One Time Password Algorithm .You can find the specification here.

Below set of images will explain you what is happening behind there.

Figure 1.1- Authentication Server Generates an key to share with the GA app.

Figure 1.2- User enter the key to GA app- No communication between server and the GA app.

Figure 1.3- Now Both sides have the shared key.

Figure 1.4 - User authentication using the TOTP.

*Note that the app and the server did not communicate by any means via network. Thus the possibility for leak the secret is minimized.

Ok now we know what happens there and it is time to get into the code and implement our own two factor authentication. For this part we will develop a simple web application to demonstrate server side authentication part.

Using Eclipse Juno and Apache Tomcat for this sample development.

1. Create Dynamic Web Application Project.

#. Add index.jsp with simple login form in the body

  <%@ page language="java" contentType="text/html; charset=UTF-8"
    pageEncoding="UTF-8"%>

*Note that checking the checkbox means its for setting up request Otherwise its verification request.AuthController will be the servlet for handling the login action.

2. Create Servlet - AuthController and override doPost() as follows.

package home.test.auth;

import java.io.IOException;
import javax.servlet.ServletException;
import javax.servlet.annotation.WebServlet;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

/**
 * Servlet implementation class AuthController
 */
@WebServlet("/AuthController")
public class AuthController extends HttpServlet {
 private static final long serialVersionUID = 1L;

 /**
  * @see HttpServlet#doGet(HttpServletRequest request, HttpServletResponse response)
  */
 protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
  // TODO Auto-generated method stub
 }

 /**
  * @see HttpServlet#doPost(HttpServletRequest request, HttpServletResponse response)
  */
 protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
  
  String username = request.getParameter("username");
  String password = request.getParameter("password");
  boolean is2faSetup  = true;
  
  if(request.getParameter("setup") == null)
  {
   is2faSetup = false;  
  } 
  
  if (username != null && password != null) {
   
   if(is2faSetup){
    // user want to set up 2fa 
    request.setAttribute("username", username);
    request.getRequestDispatcher("/SetUpController").forward(request,
      response);
   }else{
   // forward to verify code    
    request.setAttribute("username", username);
    request.getRequestDispatcher("/auth.jsp").forward(request,
      response);
   }
       
  } else {
   
   request.setAttribute("error", "Unknown user, please try again");
   request.getRequestDispatcher("/index.jsp").forward(request,
     response);
  }
 }

}

3. Create Class call GoogleAuthenticator.This is the Server side class for Google Authenticator's TOTP generated.Thanks to this excellent blog I can reuse most of the code from his blog post for this.

import java.security.InvalidKeyException;
import java.security.NoSuchAlgorithmException;
import java.util.Arrays;
import java.util.Random;

import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;

import org.apache.commons.codec.binary.Base32;

/**
 * Java Server side class for Google Authenticator's TOTP generator 
 * Thanks to Enrico's blog for the sample code:
 * @see http://thegreyblog.blogspot.com/2011/12/google-authenticator-using-it-in-your.html 
 * 
 * 
 * @see http://code.google.com/p/google-authenticator 
 * @see http://tools.ietf.org/id/draft-mraihi-totp-timebased-06.txt
 */
public class GoogleAuthenticator {
  
 // taken from Google pam docs - we probably don't need to mess with these
 final static int secretSize = 10;
 final static int numOfScratchCodes = 5;
 final static int scratchCodeSize = 8;
 
 int window_size =3;  // default 3 - max 17 (from google docs)
 
 /**
  * set the windows size. This is an integer value representing the number of 30 second windows we allow
  * The bigger the window, the more tolerant of clock skew we are. 
  * 
  * @param s window size - must be >=1 and <=17.  Other values are ignored
  */
 public void setWindowSize(int s) {
  if( s >= 1 && s <= 17 )
   window_size = s;
 }

 /**
  * Generate a random secret key. This must be saved by the server and associated with the 
  * users account to verify the code displayed by Google Authenticator. 
  * The user must register this secret on their device. 
  * @return secret key
  */
 public static String generateSecretKey() {
  // Allocating the buffer
  byte[] buffer = new byte[secretSize + numOfScratchCodes* scratchCodeSize];

  // Filling the buffer with random numbers.
  // Notice: you want to reuse the same random generator
  // while generating larger random number sequences.
  new Random().nextBytes(buffer);

  // Getting the key and converting it to Base32
  Base32 codec = new Base32();
  byte[] secretKey = Arrays.copyOf(buffer, secretSize);
  byte[] bEncodedKey = codec.encode(secretKey);
  String encodedKey = new String(bEncodedKey);
  return encodedKey;
 }

 /**
  * Return a URL that generates and displays a QR barcode. The user scans this bar code with the
  * Google Authenticator application on their smartphone to register the auth code. They can also manually enter the
  * secret if desired
  * 
  * @param user   user id (e.g. fflinstone)
  * @param host   host or system that the code is for (e.g. myapp.com)
  * @param secret the secret that was previously generated for this user
  * @return the URL for the QR code to scan
  */
 public static String getQRBarcodeURL(String user, String host, String secret) {
  String format = "https://chart.googleapis.com/chart?chs=200x200&chld=M%%7C0&cht=qr&chl=otpauth://totp/%s@%s%%3Fsecret%%3D%s";
  return String.format(format, user, host, secret);
 }

 /**
  * Check the code entered by the user to see if it is valid
  * @param secret  The users secret. 
  * @param code  The code displayed on the users device
  * @param t  The time in msec (System.currentTimeMillis() for example)
  * @return
  */
 public boolean check_code(String secret, long code, long timeMsec) {
  Base32 codec = new Base32();
  byte[] decodedKey = codec.decode(secret);
   
  // convert unix msec time into a 30 second "window" 
  // this is per the TOTP spec (see the RFC for details)
  long t = (timeMsec / 1000L) / 30L;
  // Window is used to check codes generated in the near past.
  // You can use this value to tune how far you're willing to go.
  
  for (int i = -window_size; i <= window_size; ++i) {
   long hash;
   try {
    hash = verify_code(decodedKey, t + i);
   } catch (Exception e) {
    // Yes, this is bad form - but
    // the exceptions thrown would be rare and a static configuration problem
    e.printStackTrace();
    throw new RuntimeException(e.getMessage());
    //return false;
   } 
   if (hash == code) {
    return true;
   }
  }
  // The validation code is invalid.
  return false;
 }

 
 private static int verify_code(byte[] key, long t)
   throws NoSuchAlgorithmException, InvalidKeyException {
  byte[] data = new byte[8];
  long value = t ;
  for (int i = 8; i-- > 0; value >>>= 8) {
   data[i] = (byte) value;
  }

  SecretKeySpec signKey = new SecretKeySpec(key, "HmacSHA1");
  Mac mac = Mac.getInstance("HmacSHA1");
  mac.init(signKey);
  byte[] hash = mac.doFinal(data);

  int offset = hash[19] & 0xF;

  // We're using a long because Java hasn't got unsigned int.
  long truncatedHash = 0;
  for (int i = 0; i < 4; ++i) {
   truncatedHash <<= 8;
   // We are dealing with signed bytes:
   // we just keep the first byte.
   truncatedHash |= (hash[offset + i] & 0xFF);
  }

  truncatedHash &= 0x7FFFFFFF;
  truncatedHash %= 1000000;

  return (int) truncatedHash;
 }
}

*Note that according to the specification Secret key must be base32 encoded one so we are using org.apache.commons.codec.binary.Base32 thus you need to add the jar to WEB-INF/lib

4.Create Servlet -SetUpController and override doPost() as follows.

package com.clault.auth;

import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.OutputStream;

import javax.servlet.ServletException;
import javax.servlet.annotation.WebServlet;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import net.glxn.qrgen.QRCode;
import net.glxn.qrgen.image.ImageType;

import com.google.auth.GoogleAuthenticator;

/**
 * Servlet implementation class SetUpController
 */
@WebServlet("/SetUpController")
public class SetUpController extends HttpServlet {
 private static final long serialVersionUID = 1L;
       
    /**
     * @see HttpServlet#HttpServlet()
     */
    public SetUpController() {
        super();
        // TODO Auto-generated constructor stub
    }

 /**
  * @see HttpServlet#doGet(HttpServletRequest request, HttpServletResponse response)
  */
 protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
   
 }

 /**
  * @see HttpServlet#doPost(HttpServletRequest request, HttpServletResponse response)
  */
 protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
 
 String username = request.getParameter("username");
  
  String secretKey = GoogleAuthenticator.generateSecretKey();
  
  request.getSession().setAttribute( "secretKey", secretKey );
  
  String s = "otpauth://totp/"+username+"?secret="+secretKey;

  ByteArrayOutputStream outs = QRCode.from(s).to(ImageType.PNG).stream();

  response.setContentType("image/png");
  response.setContentLength(outs.size());
  
  OutputStream outStream = response.getOutputStream();
  
  outStream.write(outs.toByteArray());
  
  outStream.flush();
  outStream.close();

 }

}

 

*Note that above I have user QR code generator library form QRGen and Zebra Crossing (ZXing) all three jars should be put in to WEB-INF/lib

5.Create a jsp call - auth.jsp

<%@ page language="java" contentType="text/html; charset=UTF-8"
 pageEncoding="UTF-8"%>
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>Auth JSP</title>

</head>

<body> 
  <form action="VerifyController" method="post">
 <table>
  <tr>
   <td>Enter Your Key</td>
   <td><input type="text" name="code"></td>
  </tr>
 </table>
 <input type="submit" value="submit">
 </form>
 
</body>
</html>

6. It submits the code  for verification to VerifyController servlet let's create it.

package com.clault.auth;

import java.io.IOException;
import java.io.PrintWriter;

import javax.servlet.ServletException;
import javax.servlet.annotation.WebServlet;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

/**
 * Servlet implementation class VerifyController
 */
@WebServlet("/VerifyController")
public class VerifyController extends HttpServlet {
 private static final long serialVersionUID = 1L;
       
    /**
     * @see HttpServlet#HttpServlet()
     */
    public VerifyController() {
        super();
        // TODO Auto-generated constructor stub
    }

 /**
  * @see HttpServlet#doGet(HttpServletRequest request, HttpServletResponse response)
  */
 protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
  // TODO Auto-generated method stub
 }

 /**
  * @see HttpServlet#doPost(HttpServletRequest request, HttpServletResponse response)
  */
 protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {

  String codestr = request.getParameter("code");
  long code = Long.parseLong(codestr);
  
  long t = System.currentTimeMillis();
  com.google.auth.GoogleAuthenticator ga = new com.google.auth.GoogleAuthenticator();
  ga.setWindowSize(5);  //should give 5 * 30 seconds of grace...
  
  String savedSecret = (String)request.getSession().getAttribute("secretKey");

  boolean r = ga.check_code(savedSecret, code, t);
  
  PrintWriter pw = response.getWriter();
  
  if(r){
  pw.write("OK");
  }else{
  pw.write("Fail");
  }
  
 }

}

Deploy the web application to tomcat.All done. It's time to test the application with Google Authenticator App.

7.1 When application deployed, go to  http://localhost:8080/GoogleAuthenticator/ and enter some user name and password. And make sure the checkbox checked before send request. If you see the AuthController code it is the parameter to identify whether this is an account set up action or just verification login.

7.2 When you submit with the checkbox checked , the server will respond QR code which holds the Your user name SecretKey

(Shared Key)

7.3 Now open up the Google Authenticator app on your phone. Select “Set up account” from the option menu.

 

7.4 Then select “Scan a barcode” from the option menu.

If you have installed QR  code reader app GA will open up it for you otherwise you have to install a QR reader app.

5. After successful scan GA will add a new account and generate a new 30 sec valid code for you.

6. Now you can verify the code using a web application , Again go to http://localhost:8080/GoogleAuthenticator/ login without the checkbox checked.

You will be redirected to code verification interface. Enter the code given by GA app.

The result will be

And check for fake number like

The result will be

Done. :)

Resources.

[1] - Eclipse Project file with all the dependencies can be found here.

References.

[1] - http://thegreyblog.blogspot.sg/2011/12/google-authenticator-using-it-in-your.html

[2] - https://code.google.com/p/google-authenticator/wiki/KeyUriFormat