Working Up

by Dwayne Phillips

Some people work on big data. The rest of us find value by working on data that is bigger than we can understand without some hand-crafted tools.

For about two years I worked on a BIG DATA project. Well, that is how it was touted. BIG DATA was a cool name that impressed people. Me? I tried not to giggle or grimace when those words were spoken and those PowerPoints were blazoned across screens in the far flung reaches of the fruited plain.

Walmart was my favorite example when persons would pull me to the side and ask for clarifications. Consider every Walmart in America and the world. That’s a big number. Consider every cash register at every Walmart. That’s a big(ger) number. Consider every customer going through every cash register at every Walmart. That big number continues to be BIGGER. Consider every purchase by every customer at … I hope by now you have the BIG DATA picture in your mind. Walmart is an example of BIG DATA. They do gazillions of transactions every second of every day. They track and analyze every one.

Walmart does BIG DATA.

We analyzed relationships among persons related to significant events. Every one of our analysts knew what questions to ask. Given a couple dozen persons and one event, the analyst could pin things on the wall, connect them with yarn, and have those cool displays like on the Hallmark Movies and Mysteries channel.

The trouble was, we didn’t have a couple dozen persons, we had 200,000 persons. We didn’t have a big enough wall and enough yarn to find the answers. And no one had the time or helpers to do the job. The data is bigger than a person can analyze.

We did BIG(er) DATA.

We solved these bigger data problems by hand crafting little tools that helped us do this and that and answer the fundamental questions that every analyst knew.

• Just show me the activities of 4 July 2020.
• Just show me the persons with college degrees.
• Just show me the telephone numbers with area code 202.
• Just show me…

The “just show me” items would move on and on until we reduced 200,000 persons to two or three persons. There was the answer.

BIG DATA? No.

BIG(ger) DATA? Yes.

Value added? Yes.

Sometimes you just need to change the topic by a couple or three letters for it all to make sense.