Help Wanted!- Data, data everywhere—and not enough people to decipher it
WSJ headline 3/11/2013
51% of surveyed IT professionals currently involved in big-data projects cited ‘lack of expertise to connect the dots’ as a reason projects fail in their organization.
No other factor was more commonly cited.
Well. This post is either going to show I am incredibly naïve or incredibly smart or incredibly stupid <and clueless>.
Everyone in business is drowning in data these days. But. Here is a newsflash. We were always drowning in data <albeit different data>. I am willing to bet a shitload of money that anyone with any business experience will agree that we had so much data crossing our desks <in the good ole days> that you could build your own great pyramid of paper if you so desired. As I scratch my head over the flurry of farcical diatribes around “big data” I can’t help but be reminded of the poem “The Rime of the Ancient Mariner”:
“water, water everywhere, nary a drop to drink.”
<Bruce translation: despite the depths and vast expanse of the ocean it can’t begin to quench our thirst>
We might say the same thing about how technology has enhanced the volume of data these days. The volume of data is almost unfathomably vast. Because of that we see thousands of articles on how to sift through the data for business advantages.
This is crazy talk. Mostly because it seems like everyone is mesmerized by the quantity of data available. Anyone with any business chops will quickly point out that anyone, throughout the history of business, has always had a quantity of data available. In fact. We almost always had too much quantity <more than they could ever use>.
The access to quantity has never been an issue.
Now <part 1>. They will also point out that part of knowing your business shit is setting up efficient/effective data gathering … so you capture the most important <and not invest gobs of energy on stuff you will never use, cannot use, do not really want to use or is just plain useless> data.
– Assessing the data available <with gobs available which gobs are most meaningful>
– Setting up a system to use the useful data <consistently trapping & tracking the useful stuff consistently saves time and effort>
– Analyzing the data <connecting the dots … instead of just showing numbers>
Now <part 3>. They will also point out that the third step in the process is often best done by someone who has no clue how the data is gathered or even needs to know exactly what data was not gathered <although they may at some point suggest gathering something that someone up the ladder had decided was unimportant>, but they know how to connect dots.
Now <part 4>. I will now point out we in business have been doing this for years.
Sure. More and different data may be available today but the schematic looks the same. It’s all about recognizing that sometimes a 7 has more value than a 9 and an ability to rearrange disparate point of information into some insightful piece of information.
Business management has always faced an obstacle when it comes to reaping the benefits of big data because they always need someone who can tell them what it all means <most often it is not simply a number means this>. Yet. It seems that because there are so many new ways to gather and track data there is a heightened awareness, and desire, to actually use all this data stuff <with the same good intentions that business had in the past — to gain a competitive edge or at least to keep up with the competition>. The game is the same we are just using different game pieces.
And here is what any business person with chops will also tell you — relying on data alone isn’t enough. This is a game of both head and gut. When you rely too heavily on data, you can become too reactive, too myopic in your thinking and miss out on what the numbers can never tell you … the why’s and the what’s and the <inconceivable to number crunchers> impractical inconsistent sometimes illogical human mind & behavior.
Data cannot tell you what to do. <Big> data can lead to small sharp insights and beget great decisions and action.
Here is a business truth <that most executives do not want to hear these days>. Data, of any size <double venti, regular venti, grande, etc.> has no value in and of itself. The true value of data is found in context.
You absolutely need a team with technical people to gather & mine the data but they need to be working together with an experienced analytical person who knows how to ‘connect dots.’ This type of person knows how to observe information, interpret information and place it in context with non-number/data stuff and explain it. And, no, that person may not be a data gwonk.
They are just good at connecting dots.
Oh. And they are good at not being blinded by the newest & nearest data point.
“Gut feel is great for everyday problems. But, it often leads us astray when we’re presented with complex streams of information. We can be blinded by the newest and nearest data point and miss the big picture.”
Nate Silver statistician & author
I don’t agree with Nate, well, he did caveat it with “can” and “often” so maybe I will give him a break. Gut feel … intuition … ability to “feel” the numbers in context … is essential in order to use the data. I do believe in what IBM calls “augmenting intuition.” That means … well … what it says. Augment, ‘in addition to’, add in as part of your decision criteria.
I do worry we are heading down a path where business people are forgetting that no amount of numbers <and data of any kind> can eliminate all decision risk. Nor can any amount of numbers <and data of any kind> insure you make the best decision.
Here is my last “Truth” of this post: data & analytics can make you equally smart & stupid.
People make smart decisions using data all the time.
People make stupid decisions using data all the time.
The only thing consistent is people. And here is where the WSJ article kind of truly went a little nutso.
This is where it all falls apart for me.
Because doing what they suggest basically means that data drives good decisions. Data all by itself. No intuition, no feel, no gut from experience, the suggestion that maybe data can make a decision for you — and they are wrong.
I become scared because I almost feel like this is a deeper dive into that business hellhole I call “responsibility free decision making” <wherein people absolve themselves of responsibility for decision by using numbers to cover their asses> with the intent to do the “safest behavior to increase return <or increase advantage>”. This is using data to make all the decisions <and they even use it to hire a person which is also kind of nuts>.
This is dancing on the head of a pin business management. More worrisome is it doesn’t teach people how to think.
It doesn’t utilize skills of existing people <who aren’t steeped in ‘Big Data” but are also not intimidated nor blinded by the newest & nearest data point> who are very good at connecting dots. And, worse, it guarantees a next business generation of “big Data decipherers” or people who use data decision making skills and have honed no intuition skills at all.
Am I suggesting “gut management” alone? Of course not. I never have. I never will.
In the 80’s we scoured computer printouts with ‘crosstabs’ and supermarket SAMI and Nielsen reports which contained reams of data point we had to make sense of. In the 2000’s we are scouring computer printouts <assuming you print out> which contain reams of data points we have to make sense of. And you did it then, as it should be done now, as part of a team to insure you didn’t get dazzled by some shiny data point.
This stuff drives me a little nuts because we all think the newest and nearest data point <oops … innovation> means that the world has turned on its head.
Some skills are just, well, good business skills. Adaptable to pretty much any new widget or innovation that mankind can create.
I know how to connect dots. I have no clue how to build systems to gather these dots. And you know what? I am not sure I have ever known. And I am not unique. There are hundreds if not thousands of Me’s out there.
Making Big Data into nice small simple learnings/conclusions. Ok. Making any data available into nice small simple learnings.
Pick a year. 2013. 1913. 1813.
The skill has always been relevant and thinking that ‘data decipherer’ is some new skill is crazy.