Introduction

Big data is all the rage and gets a lot of pixel time in blogs and news sites. As you may know, "big data" is a term that describes particularly large sets of data, including unstructured and semi-structured data, and the tools and processes that are used to manage it.

Although big data might be the topic getting all of the attention today, CIOs and those responsible for data and business intelligence should not divert their attention from what I call “small data.” This is the structured data that is used in daily operations to achieve business goals.

Small data does big things

The data that organizations capture in their routine operations is critically important to that eventual success of that organization. It’s used to either initiate or accomplish any number of business processes and provides context and information for operational decisions that need to be made by the organization.

My background lies primarily in higher education, so let me provide some examples from my experience that may help illustrate this point.

Admissions departments capture all kinds of demographic and background information about prospective students, including name, address, test scores (ACT or SAT, for example), high school GPA, and athletics and other extra-curricular participation. Most of this information is used at some point to make decisions. For example, ACT or SAT scores might be used to help place students in appropriate math courses. The high school GPA might be used in a process to determine whether or not the prospect will be admitted to an honors program. This isn’t big data, but it’s still critical.

However, even though people may not consider college admissions data to be “big data”, it can be mined to improve the college’s bottom line, and particularly retention, one of two key revenue drivers for most colleges and universities. Retention is a measure of the number of students who choose to stay at the college year after year. As is the case in any business, keeping an existing customer is less expensive than finding a new one. In higher ed, there is a growing market for retention intelligence whereby student data is mined (privately, by the institution) to look for patterns that might prove to be indicators for success or failure of that student. This is the beginning of “big data” as the term has come to be defined and can help institutions identify strategic initiatives that might be undertaken to improve the retention metric.

I imagine that most verticals have similar opportunities using different kinds of operational data.

Operational data is the catalyst to success

In every organization, data has a lifecycle. As admissions data becomes information about enrolled students, that data eventually becomes information about the alumni base. But all along the way, additional data is added that carries with it additional value for the organization. while the individuals are students, new activity information is added to operational data. By the time that data gets to the next part of its lifecycle, the alumni and fundraising office, it should be laden with information that can be leveraged by that office to maximize the value of the efforts of the staff.

For example, when data regarding student activities is captured correctly and consistently, the fundraising office can create affinity groups of alumni with similar interests that enhance the office's efforts. For instance, phone outreach programs are more effective when current students with particular interests contact alums that participated in similar activities while they were in college.

However, these efforts are most effective when the institution remains committed to data quality and consistency, so even while big data projects are undertaken, the tactical nature of small data demands that a significant focus remain on it.

Big data can help open new opportunities for the willing

Whereas I see small data as helping organizations achieve tactical goals, I see big data as helping organizations in a strategic way in charting new paths and identifying new opportunities. Again, let’s turn to higher education and retaining students. I indicated that mining admissions and student data to search for unseen patterns that could identify retention risks and retention positives can have a significant impact on the bottom line if it’s approach from a strategic perspective and the opportunity is not squandered.

For example, a tactical institution would identify retention risks and then instruct its admissions department to no longer recruit students that show high likelihood of negative retention. A forward-thinking, strategic institution, on the other hand, would look at the same data set and, rather than saying, “We’ve found the problem!” would say, “How do we solve this problem? Why are students that match these particular factors not remaining with us? What steps, services, and processes do we need to put into place to improve the retention metric among this population?”

At this point, the institution, rather than instantly reacting to what appears to be a negative, would do a cost/benefit analysis of proposed solutions to determine the best step to take. Is the cost worth addressing the retention issue or are funds better directed to other positive revenue endeavors? This is a strategic way to use the data rather than tactically responding by simply eliminating a potential candidate pool.

The key here is that the organization needs to be willing to reflect upon itself and admit that it might have weaknesses to be addressed. Without the willingness to change for the positive, organizations can’t make strategic decisions no matter how much “big data” they analyze. It’s only those with a truly open and reflective culture that can fully capitalize on the opportunities presented by data both big and small.

Action Item: While ensuring that the company culture is one that can embrace change to the level that might be needed to implement changes that can drive it to new levels, CIOs also need to ensure that their organizations are maximizing their use of “small data” or operational data in order to maximize the effectiveness of business processes. At the same time, in willing organizations, CIOs need to begin big data efforts—even on a small scale—that make sense for their industries and use the data in an attempt to identify potential business opportunities that might make sense or to identify organizational weaknesses which, once corrected, might be just as good as starting a whole new product line.

Footnotes: