Sometimes I need to make payments by credit card and the only trustworthy Internet access available is my Android phone. That is a problem because the MBNet site is not designed for small touch screens. It has a virtual keyboard that is a mess to use on a touch device. Not satisfied with this situation, I decided to create an Android application that could be used as a front-end for the Web site. The result can be seen in the following pictures.
Downloading the application
Warning: When you download applications from unknown sources, you are exposing yourself to a risk. Before using this application, think about the following risks:
- Maybe this application sends your user name and password to some random hacker?
- Even if you trust me, maybe the application has a bug that could expose your password?
- This site might have been compromised and the application has been replaced with spyware?
In general, instead of downloading the application, you should download the source code, review it and then compile it yourself. In any case, I will not be held responsible if something happens to you after installing it.
If you are still interested, you can download the MBNet application here.
Implementation details
Since MBNet does not provide a publicly accessible API, which is understandable due to the nature of the service, I had to resort to Web scraping to implement the application. Web scraping consists of having a program interact with a Web site as if a real user was accessing it. This technique has some drawbacks. One of them is that even minor changes to the site can cause the application to break. Since there's no other way to do it, we will have to live with these potential problems.
A web scraper can be implemented in various ways. Some of them include:
- performing HTTP requests to the site;
- embedding a Web browser in the application.
After trying both techniques, I found that the Web browser solution was simpler because a Web browser already provides cookie management and javascript evaluation. Cookie handling is handy because we don't have to care about them while sending requests. Javascript was necessary because this Web site relies on client-side script to protect the user name and password of the user.
The Android platform provides the WebView control to display HTML pages. Opening a Web page is as simple as invoking loadUrl on the WebView:
webview.loadUrl("https://www.mbnet.pt/mbnetAut.html");
By default, javascript is disabled. We need to enable it using the following code:
webview.getSettings().setJavaScriptEnabled(true);
Interacting with the Web page
In order to do something useful, we need to be able to interact programmatically with the page. We will want to fill fields, submit forms and retrieve text. Usually, Web browser controls provide some way to manipulate the DOM and to invoke Javascript code on the page. WebView provides none of these. It allows to register objects that can be invoked from Javascript, but not the other way around. That is a problem because the pages that we are loading know nothing about our objects, therefore they won't invoke their methods.
Fortunately, it is possible to use a trick to work around the limitations of the WebView. If we tell the WebView to navigate to a javascript: url, the browser will execute the code after the semicolon in the context of the current page. This is how bookmarklets are implemented. This allows to do the following:
webview.loadUrl("javascript:eval('document.write(\"hello world\")')");
However, if we want to execute non-trivial code, this approach becomes hard to manage. To fix that I registered a javascript interface with a method that returns the actual script to execute, and the navigate code simply invokes that method and evaluates the resulting script. The following snippet illustrates that method.
webview.addJavascriptInterface(new Object() {
public String getScript() {
// The code can be obtained from a resource or whatever
return "document.write('hello world');";
}
}, "itf");
webviewloadUrl("javascript:eval('' + itf.getScript())");
Conclusion
After finding out how to communicate with the page inside the WebView, it proved a viable solution for implementing Web scraping. I encourage you to read the source code for more details.
License
This program, along with its source code, is free software. You can use it under the terms of the GNU GPL license. For more information please consult the license.txt file inside the source distribution.