A simple idea to improve database scalability

September 10th, 2011 No comments »

Scalability is a big issue. How scalable is a database based system ? Can it cope with increasing demand ? Will the website / system / whatever crash when increasing numbers of punters push increasing numbers of queries into your table ?

Excuse me for being an old cynic, but is this perchance an excuse for vendors to sell even more expensive databases to the customer ?  Specialist database suppliers  sometimes have products which are not nearly as fast or “scalable” as their salespersons like to pretend ( notice how I have carefully not used any names.  Lawyers are generally unpleasant, pompous, self-regarding and extremely expensive. )

So, what to do ? Well, here’s a cheap and cheerful idea.  ( Please let me know if any of you furiously disagree, and I will proceed to carefully consider and then ignore your comments. )

Storage space is dirt cheap these days. A basic web hosting deal can supply 150 GB to 200 GB or some other ridiculously huge amount of space, for less than the price of a dinner in an overpriced London eatery.

Duplicate your table and its content. Better still, clone it three, four or more times.  Assuming you have users who login to access the data, sort them according to their ID number ( or whatever the unique identifier is ). Then, redirect them to table 1, table 2,table 3 etc according to their ID.  Remember, every table is identical, with all the same field names, data etc. Only the table names differ.  ID data is stored in a session object on login. So, on login :

<?php

session_start();

$ID = $_POST['ID'];

$ID = trim($ID);

$_SESSION['Id'] = $ID;

?>

or, something along those lines.

At the query processing stage, assuming you have between 200 and 300 unique users :

<?php

if ($_SESSION['Id'] <  101) {

$SearchQuery = “SELECT ID, foo, bar FROM table1 WHERE foo = \”par1\”  ;

}

elseif ( ($_SESSION['Id'] > 100) && ($_SESSION['Id']  <  201)) {

$SearchQuery = “SELECT ID, foo, bar FROM table2 WHERE foo = \”par1\”  ;

}


else    {

$SearchQuery = “SELECT ID, foo, bar FROM table3 WHERE foo = \”par1\”  ;

}

Then, continue as normal :

$link = mysql_connect(’localhost’, ‘username’, ‘password’) or die(’Could not connect: ‘ . mysql_error());

mysql_select_db(’database’) or die(’Could not select database’);

$result = mysql_query($SearchQuery) or die(’Search Query failed: ‘ . mysql_error());

etc.

Same query name, returns same result data from different tables.  You are effectively spreading the load among different tables according to your users’ ID.

Simple.

Obviously all the above is done using php and MySQL.  Nothing fancy or expensive.

BTW -  I strongly recommend NavicatMySQL ( which I use -  there may be similar products out there, that I don’t use………..).

Navicat is quite affordable and makes the tasks of manipulating MySQL tables, data, exporting, importing, dumping, type conversion etc a breeze.

Hope this helps delay the day when you need to fork out for a really expensive database system   ;-)

PS    Please don’t tell me that it won’t work for an “enterprise” level system. 

Firstly – there are different sizes of enterprise,

secondly - I am perfectly aware of the bleedin’ obvious ( to quote Mr Basil Fawlty ),

thirdly - “enterprise” level systems routinely crash when a large number of users try to access them.

As any regular user of British government IT systems will tell you.






Storing Dates in MySQL using PHP

August 29th, 2011 3 comments »

If you just want to store date information in MySQL, with no associated manipulation, the easiest way is to declare it as data type int . ( I am assuming all scripting is php ).   As you are probably aware, Php and MySql format date data differently and so conversions are required between the two. If you are only storing data, all this extra work is completely unnecessary.

We use the Php time() function, which is UNIX based and gives us the number of seconds elapsed since Jan 1, 1970, at the moment the function is called.   So, for example, to check if an annual subscription is still valid :

<?php

$thistime = time();

/*  you want to check if annual subscription is still valid; ie less than 1 year has elapsed since  time was logged in database  */

$chkTime = $loggedTime + 31557600;

/*    31557600 is number of seconds in 365.25 days  – allows for leap years   */

if ($chkTime < $thistime)  {

//    allow access – subscription still valid !

}

else  {

//  deny access – client has to pay for another year !

}

?>

And that’s it !   OK, it’s only  good for simple date storage and checking, and when you don’t need to display date information. However, it’s useful in a lot of situations and doesn’t require any complex conversion functions.

Hope this is useful.


Adservers slow down webpages

August 27th, 2011 7 comments »

This isn’t really very great news but the adverts that clog most commercial websites slow down webpage loading to a horrendous degree.  I have even had browsers ( especially Chrome ) crash while waiting for an adserver to download its unwanted dross.

In addition advertisements served by independent advertising agencies are a real source of viruses and trojans, as this article describes.

Even when there is no virus, per se, just waiting for the completely unwanted ads to load causes frustration and annoyance ( to me at least ! )

Luckily help is at hand with the new version of Adblock called Adblock Plus. This is a completely free download which I have just installed on Firefox. I must say it’s revelation. Pages load just so much faster and now consist of pure content rather than all the distracting peripheral animations and visual junk that clog up so many sites. I used to have the old Adblock which I lost, uninstalled, somewhere along the way, but this one is just so much better.

However, web publishers who rely on the get-web-traffic-to-sell-ad-space-to-generate-revenue business model may not be too pleased with everyone installing ad blocking software.

But that, my friends, is the world – a cold hard place, especially for many old media newspapers which are struggling to make ends meet in the new environment.

P.S.  To affiliates who have ads “served” by their principal, I would suggest hard coding hyperlinks to the relevant sites, as well as having ads from servers including images etc.  Hyperlinks aren’t blocked by adblocking add-ons, whereas served advertisements are, and shouldn’t be relied on.

Politicians shoot the messenger ( as usual )

August 12th, 2011 No comments »

In the wake of the London and UK riots our governing classes ( geniuses, all of them ) have decided that the problem is largely because of  ”social networks” and that access to them may be blocked . Quite how they propose to do this, they don’t specify. There were even calls by MPs to close down Twitter and other networks completely, or block access to them.

Apart from the fact that this is a stunt that the regimes in Tehran or Pyongyang might dream up – it is ultimately futile.  If rioters contact each other by telephone landline, will they ban telephones ? If potential rioters communicate by handwritten notes, do they propose to ban pens and paper ? Perhaps Louise Mensch should do everyone a favour and keep her idiotic opinions to herself.

Once again our ruling classes have shown themselves to be clueless buffoons.

Good news for Android Developers ?

March 17th, 2010 No comments »

Here is good news for established ( and budding ) Android developers. It seems that the Apple iPhone is no longer the only game in town !

I predict that Android will become an increasingly important development environment, possibly overtaking proprietary Apple software at some point.

Well, I got that off my chest ………….

How will Twitter’s @anywhere work ?

March 16th, 2010 No comments »

At the SXSW conference in Austin Texas yesterday, Twitter announced the new @anywhere “platform” as CNET called it, although it is apparently just a few lines of javascript.  A way of putting Twitter links and data on to partner sites, and enable users to follow posts of their favourite journalists, for example, from a newspaper website, without going to the Twitter site, it seems to be a way of extending Twitter’s reach as well as ensuring that a user is following the writer they wish to, and not an “impersonator”.

On Twitter itself, it may not be obvious who a Twitter “tag” belongs to, so it seems to be good idea from that point of view at least.  Early adopters or launch partners include Advertising Age, Amazon, Bing, eBay, Citysearch, Digg, Huffington Post, Meebo, MSNBC, The New York Times, Salesforce, Yahoo and YouTube. In other words, some very big boys indeed.

However as Wired points out, audience reaction was muted and less than enthusiastic.  Perhaps it’s because they didn’t really see it as such a big deal from the user’s point of view, but for Twitter it could become a potential source of revenue. The question that remains is, how much can you charge for what is essentially just a few lines of javascript ?  Apparently the service is free at present for the early adopters, but the business model is still developing.

How it will develop in the future and whether it will be available / affordable for the small website owner remains to be seen.

Can Robots solve the problem of an ageing population ?

March 2nd, 2010 1 comment »

Japan has an even worse problem than Western Europe with an ageing population and a shrinking productive workforce. The Japanese however seem to be pinning their hopes on little guys like this :

Would you like this chap to vacuum the carpet ?

Would you like me to vacuum the carpet ?

Now the idea of automata is not a new one. Numerous writers in ancient Greece described them including Hesiod. Indeed Greek technology had its own god, Hephaestus who is said to have created the original Terminator, Talos, the man of bronze, not a guy to mess with. And while we’re on the subject, the Greeks seem to have made the first computer, the  Antikythera mechanism.

Automata, as toys, were known in 17th and 18th century France as well, although they weren’t quite as advanced or versatile as the Honda Asimo.

Although still a prototype, in the not too distant future fellows like this one may be vacuuming the carpet, doing the dishes, loading the washing machine, walking the dog and cleaning the windows.

If you’ve watched the Terminator movies – perhaps you’ll think it’s not a good idea to make them too intelligent or put them in charge of nuclear warheads !

Interesting background info on Artificial Intelligence / robotics :

Views Into The Chinese Room is a fascinating collection of essays by philosophers and leading researchers in the field of AI and robotics.

Blondie24 describes how a computer program using an evolutionary algorithm, became a near championship level draughts ( checkers ) player.

For those of you who would like to try building their own robot how about :

Robot Building for Beginners, 2nd Edition (Technology in Action) by David Cook

Absolute Beginners Guide to Building Robots (Absolute Beginner’s Guides) by Gareth Branwyn

PIC Robotics: A Beginner’s Guide to Robotics Projects Using the PIC Micro (TAB Electronics) by John Iovine

A Simple Way of Creating a PHP Web Service Client with WSDL

February 8th, 2010 3 comments »

Web services !  Don’t ya love ‘em ? Everybody is talking about it, just like sex at high school. But how many are actually doing it ? ( just like …………ok you get the picture )

The problem is that when it comes to writing a web service client, even in something like php, it can get HORRENDOUSLY complex.  In order to get any meaningful content out of  the  SOAP response ( and I’m talking about SOAP here ) everyone, from what I can gather, talks about parsing. Now parsing the XML of what comes back in the response can be dreadfully complex. It can also be unavoidable if you don’t know precisely what’s being returned.

OK, I’m assuming you DO KNOW the structure of the XML response. And how do you know it ? Because you are accessing a web service via a request that is defined in a pre-existing WSDL document.

Many standard IDES like Eclipse Galileo, for example, already integrate a Web Service Explorer with which you can access a given WSDL document, select the service you want from the list and create a request on the fly, fill in some request data, your access password etc and click send. You instantly have the response code in the response window.

And that’s what we analyse.

Essentially my simple method consists of turning the returned SOAP into a well-formed XML document. In other words getting rid of the the SOAP headers and envelope which just complicate matters.

You need to download nusoap because  that is a prerequisite. The top of your document will have :

require_once(’nusoap.php’);

You define an endpoint as the URL of your given WSDL file as in :

$endpoint =  “” http://www.someservice.com/myservice”;

You define your client as :

$client = new nusoap_soapclient($endpoint, true);

You build your SOAP header, the details of which I won’t go into here, and you call the required service in the usual way with the $client->call(); method.

Define your response as in :

$feed =  $client->response;

This is in essence a long string. I can then use PHP string cutting and manipulating functions to separate the SOAP envelope header as well as end tags from the important “meat” of the response that contains the data and then “rebuild” the string as well formed XML by concatenating the resultant string to an opening bit like :

$XMLHeaderStr = “<?xml version=\”1.0\” encoding=\”UTF-8\”?>”;

Ditto for the end tags.  Your resultant string will be at the end one long XML document that is “well-formed” (we hope) and thus it becomes eliible for use with the simplest XML processing function in PHP, namely :

$xmlString = simplexml_load_string($NewFeed);

That’s it, in essence. simple_load_string() is a much easier proposition for manipulating in your PHP document and it really works !

To limit length of posts I have only outlined the method here. For further information you can email me.

Recommended Reading :

Web Services: Principles and Technology

Sams Teach Yourself Web Services in 24 Hours

Web Services: Concepts, Architectures and Applications (Data-centric Systems and Applications)

Programming PHP

Java Web Services: Up and Running


How does the Google myTouch stack up against the iPhone 3G

February 3rd, 2010 10 comments »

Well, how do they compare ?

Watch this video ………….

The Google myTouch compares very favourably and the point is that it is still very early days for phones using the Android OS.  Apple software is proprietory with its associated “lock-in” whereas Android is the OS of choice of the Open Handset Alliance whose members can be seen here.

2010 is shaping up to be a year of massive predicted expansion and competition in the world Smartphone market and although it is still early days for Android based phones, I predict we will be hearing much more about them.

Watch this space as they say……

Get Adobe Flash playerPlugin by wpburn.com wordpress themes