The MBNet service
Those living in Portugal will be familiar with the
MBNet service. It
is a service that allows to use a
Portuguese debit card to make payments in sites that accept credit
cards only. In practice, you login with a user name and password,
enter a maximum amount, and it generates a virtual credit card
number that can be used for a single payment for up to the amount
you specified. This service is very useful and I use it for most of
the payments I make on the Internet.
Sometimes I need to make payments by credit card
and the only trustworthy Internet access available is my Android
phone. That is a problem because the MBNet site is not designed for
small touch screens. It has a virtual keyboard that is a mess to
use on a touch device. Not satisfied with this situation, I decided
to create an Android application that could be used as a front-end
for the Web site. The result can be seen in the following
pictures.
Downloading the application
Warning: When you download
applications from unknown sources, you are exposing yourself to a
risk. Before using this application, think about the following
risks:
- Maybe this application sends your user name and password to
some random hacker?
- Even if you trust me, maybe the application has a bug that
could expose your password?
- This site might have been compromised and the application has
been replaced with spyware?
In general, instead of downloading the application, you should
download the source code,
review it and then compile it yourself. In any case, I will not be
held responsible if something happens to you after installing
it.
If you are still interested, you can download the MBNet application
here.
Implementation details
Since MBNet does not provide a publicly accessible API, which
is understandable due to the nature of the service, I had to resort
to
Web
scraping to implement the application. Web scraping consists of
having a program interact with a Web site as if a real user was
accessing it. This technique has some drawbacks. One of them is
that even minor changes to the site can cause the application to
break. Since there's no other way to do it, we will have to live
with these potential problems.
A web scraper can be implemented in various ways. Some of them
include:
- performing HTTP requests to the site;
- embedding a Web browser in the application.
After trying both techniques, I found that the Web browser
solution was simpler because a Web browser already provides cookie
management and javascript evaluation. Cookie handling is handy
because we don't have to care about them while sending requests.
Javascript was necessary because this Web site relies on
client-side script to protect the user name and password of the
user.
The Android platform provides the
WebView control to display HTML pages. Opening a Web page is as
simple as invoking loadUrl on the WebView:
webview.loadUrl("https://www.mbnet.pt/mbnetAut.html");
By default, javascript is disabled. We need to enable it using
the following code:
webview.getSettings().setJavaScriptEnabled(true);
Interacting with the Web page
In order to do something useful, we need to be able to interact
programmatically with the page. We will want to fill fields, submit
forms and retrieve text. Usually, Web browser controls provide some
way to manipulate the DOM and to invoke Javascript code on the
page. WebView provides none of these. It allows to register objects
that can be invoked from Javascript, but not the other way around.
That is a problem because the pages that we are loading know
nothing about our objects, therefore they won't invoke their
methods.
Fortunately, it is possible to use a trick to work around the
limitations of the WebView. If we tell the WebView to navigate to a
javascript: url, the browser will execute the code after the
semicolon in the context of the current page. This is how bookmarklets
are implemented. This allows to do the following:
webview.loadUrl("javascript:eval('document.write(\"hello world\")')");
However, if we want to execute non-trivial code, this approach
becomes hard to manage. To fix that I registered a javascript
interface with a method that returns the actual script to execute,
and the navigate code simply invokes that method and evaluates the
resulting script. The following snippet illustrates that
method.
webview.addJavascriptInterface(new Object() {
public String getScript() {
// The code can be obtained from a resource or whatever
return "document.write('hello world');";
}
}, "itf");
webviewloadUrl("javascript:eval('' + itf.getScript())");
Conclusion
After finding out how to communicate with the page inside the
WebView, it proved a viable solution for implementing Web scraping.
I encourage you to read the source code for more details.
License
This program, along with its source code, is free software. You
can use it under the terms of the GNU GPL license.
For more information please consult the license.txt file inside the
source distribution.