Working Up

by Dwayne Phillips

Big data? What we usually do is automate work on data that is too big to work manually.

I used to work on a project where the project leader would tell people, “We work big data.”

I was frequently asked (later after the project leader left the room), “What is big data?”

Pause to reflect. Consider WalMart. They have thousands of stores in America and other countries. They have many cash registers in every store. They have many customers at every cash register in every store in every country, and we have many, many, many individual items bought.

WalMart collects all that. Every single item sold is stored and analyzed. They don’t throw away any of it (those in government could learn many lessons here). That is big data. I don’t know how many individual items WalMart sells in a day or a minute, but it is a big number and that is big data.

The rest of us are not WalMart or Apple or Amazon or any other outfit that has many, many transactions per minute. The rest of us are trying to do our jobs with the data and questions that we have. The rest of us struggle to finish the work by the end of the day so we can go home and eat dinner.

The rest of use work with too big data. It is too big for the rest of us to do manually. The rest of don’t have the resources to automate the work so that we can do it in minutes instead of days. The rest of us just try to get by. The data is too big.

Solutions? Yes, there are many. The rest of us are happy to hear about them and hoping that someone will bring them to us so we can handle the data that is too big and we can go home on time and eat dinner.