Big Data and the Soviet Ghosts

In a 2009 CNBC, Maria Bartiromo asked Google’s Eric Schmidt if people should treat Google like their best friend. His reply reflects the ethos of the modern software world. He said that you shouldn’t do anything that you wouldn’t want to get on the internet. In other words, everything people do should be recorded. The modern character of the new software world is Big Data. Big Data and the machine learning tools bring a ghostly vision to the software world that Soviet technocrats, who ran the Soviet Union, experimented with during the 60s and 70s. That we can record everything in society, and develop a rational way to understand society, for the purpose of guiding it to new prosperity.

Here is a clip from the documentary Pandora’s Box:

In the clip, it shows how information from all of Soviet society is fed into computer systems in Gosplan. The belief being that with all of this data, you can use models to determine if society is following the plan. It was a tool the technocrats used to control society in an efficient way. Without the computer systems, all of Soviet society would have to be employed to crunch the economic data mechanically.

Big Data  promises the same dream, but comes at it from a different side. Instead of developing rational models and applying them to large amounts of data, machine learning is employed to bubble up predictive models and to find connections between disparate features automatically. But the goal is still the same, to make sense of vasts amount of data for the purpose of control. The control the Soviet Union used was straight-forward top down. The control modern western industry employs is indirect and bottom up. This is what advertising companies do, and the dream that Google and friends sell to them.

But just like the Soviet Gosplan, the Big Data software world requires enormous amounts of centralization. Big Data requires enormous star networks that funnel every piece of data through them. The dream of the internet being a distributed system has been crushed by the client-server architecture of the Web. And because this centralization is so valuable, huge investments were made in the mid to late 90s that led to the Dot Com Bubble, and from it the internet giants we have to day such as Google, Amazon, Ebay, and Yahoo.

Just as the centralized Soviet economy and computer systems produced gross absurdities in Soviet society, the centralization of information has produced vast obscurities in the lives of this generation. As demonstrated by luke warm reaction to the recent revelations about the NSA, to the Great Recession and the computerization of trade, to the perverse incentives Facebook and Twitter provide for exhibitionist behaviours, to the death march of consumerism despite the hard evidence that global warming will lead to a miserable existence for hundreds of millions of people within decades.

The private life is dead, so that google and facebook can help others sell you things you don’t need to keep the economy humming. And it all works too! But in a much more hidden, mysterious way than Gosplan could have ever hoped.

Bonus Clip describing the obscurities that developed in Soviet society

10 thoughts on “Big Data and the Soviet Ghosts

  1. This article is clearly from an NSA shill. “private life is dead”, “keep the economy humming”, “And it all works too!” This is the shit you want to feed the masses? This is what you think people should believe?

  2. The Soviets weren’t the only ones to envisage a centralised system of economic management.

    Stafford Beer was the architect of the (arguably) more obscure Cybersyn system in Chile, devised under the socialist Allende government. Given Beer’s close association with cybernetics and his reputation stemming from it, it’s perhaps understandable that he would perceive a national economy as being one that could be subject to changes in the inputs and outputs of it in order to control it.

    https://en.wikipedia.org/wiki/Project_Cybersyn

    • I can’t agree with this. Stafford Beer’s Cybersyn was not a ‘centralised system of economic management’ in the sense of concentrating decision making at the top. The whole approach was to avoid uncontrollable amounts of information flooding up to the higher levels when decisions could be handled at lower levels. That’s why his Viable System Model, which underlies Cybersyn, is very explicitly recursive.

      All states have decision making functions at the centre in the sense that they have central banks, government departments in charge of the economy, etc. Beer accepted that, and built those into his system in attempt to help them to work on more realistic models of the economy. Of course if you don’t like governments then you won’t want them to work better, but that does not make Beer’s system centralised. On the contrary, you could see it as a model for devolution.

      By the way, if you want to know about Stafford Beer then don’t trust Curtis’ films, which are uniformly (and IMHO unreasonably) negative about cybernetics in general. Read his books instead.

  3. I’m curious what you’re referring to by “the hard evidence that global warming will lead to a miserable existence for hundreds of millions of people within decades”? I don’t dispute that emitting carbon dioxide warms the atmosphere, but it doesn’t necessarily follow that this will inevitable lead to misery for hundreds of millions of people. The projections that I’m aware of from credible sources about the potential damage from global warming are more along the lines of “in a hundred years the world will only be seven and a half times richer than it is today instead of eight times richer”, so I’m genuinely curious what you’re basing this prediction on.

  4. Hi! Someone in my Myspace group shared this website with us so I came to look it over. I’m definitely loving the information. I’m bookmarking and will be tweeting this to my followers! Great blog and terrific design and style. dccdeacaeefg

Leave a reply to Dai Griffiths Cancel reply