Spy chiefs are already spending hundreds of millions of pounds on a mass internet surveillance system, despite Jacqui Smith's announcement earlier this week that proposals for a central warehouse of communications data had been dumped on privacy grounds.
The system - uncovered today by The Register and The Sunday Times - is being installed under a GCHQ project called Mastering the Internet (MTI). It will include thousands of deep packet inspection probes inside communications providers' networks, as well as massive computing power at the intelligence agency's Cheltenham base, "the concrete doughnut".
Sources with knowledge of the project said contacts have already been awarded to private sector partners.
One said: "In MTI, computing resources are not measured by the traditional capacities or speeds such as Gb, Tb, Megaflop or Teraflop... but by the metric tonne!.. and they have lots of them."
The American techology giant Lockheed Martin is understood to have bagged a £200m deal. The BAE-owned British firm Detica, which has close links to MI5 and MI6, as well as to GCHQ, has also been signed up to help on MTI.
A spokeswoman for GCHQ said the agency does not comment on individual contracts. "GCHQ works with a broad range of industry partners to deliver a complex portfolio of technical projects," she said. Detica also declined to comment, and Lockheed Martin did not return calls.
Sources said MTI received approval and funding of more than £1bn over three years in the October 2007 Comprehensive Spending Review. GCHQ, like MI5 and MI6, is funded out of the opaque Single Intelligence Account. For 2007/8 the planned budget for the three agencies was over £1.6bn.
GCHQ began work on MTI soon after it was approved. Records of job advertising by the agency show that in April 2008 it was seeking a Head of Major Contracts with "operational responsibility for the ‘Mastering the Internet’ (MTI) contract". The new senior official was to be paid an annual salary of up to £100,000.
The advertisment also indicated that the head of Major Contracts would be in charge of procurement on MTI and be expected to forge close links with the private sector.
According to sources, MTI is a core piece of the government's Interception Modernisation Programme (IMP). On Monday of last week, the Home Secretary Jacqui Smith announced that under IMP, rather than build a central warehouse, responsibility for storing details of who contacts whom, when and where will be imposed on communications providers.
The news was welcomed by privacy advocates and civil liberties campaigners, but sources described it as a "side show" compared to the massively increased surveillance capability that MTI will deliver. It will grant intelligence staff in Cheltenahm complete visibility of UK Internet traffic, allowing them to remotely configure their deep packet inspection probes to intercept data - both communications data and the communication content - on demand.
Shami Chakrabarti, director of Liberty, said: "We opposed the big brother database because it gave the state direct access to everybody’s communications. But this network of black boxes achieves the same thing via the back door."
GCHQ's spokeswoman said: "GCHQ does not discuss 'how' we use data, as this may lead to revelations about our capability which damage national security.
"GCHQ is constantly updating its systems in order to maintain and renew its capability."
Advocates of MTI and IMP say they are essential if intelligence agencies are to maintain their capability to monitor terrorist and other criminal networks.A Home Office consultation on the storage of communications data is now open. Meanwhile, work and spending on the all-seeing system to intercept and retrieve it is already underway.