Nightingale HQ Blog

Stay up to date with our most recent news and updates

    Do you really need big data to start using data science?

    Published on 21-Oct-2019 10:13:16


    All businesses generate data. Even the smallest business has access to hundreds, if not thousands, of interesting data points that they could explore. But it is not uncommon for business owners to look over their shoulders at the giants and their Big Data monoliths, and wonder if they should hold back until they have 'enough data' before they start to analyse it. In their article for Harvard Business Review1, Thomas Redman and Roger Hoerl argue on the contrary; the time to start your data science project is now.

    Big data projects are expensive, time-consuming, and - by some estimates  - carry a shockingly low success rate. Investing in these big data projects is unwise unless you have a strong foundation of data science competency and a culture of appreciation for AI and its benefits2. Redman and Hoerl suggest that instead of waiting for big data to be available, you implement a series of "small data" projects, to build this foundation and tap into the benefits of data analysis right away.

    The benefits of these small data projects are more numerous than you may expect. Redman and Hoerl cite a much higher success rate, lower costs because of smaller teams and less time requirements, and - at the bottom line - annual financial gain of $10,000 - $25,000. Perhaps more key than any of these benefits though, is the impact on company culture. Involving your staff in small analytical projects using builds skills, confidence, and appreciation for the benefits of automation and AI. "Plus, they are fun!", say Redman and Hoerl.

    What should your first small data project be? Ask your staff what would benefit them - what do they want to know? Identify a business process that you want to streamline, automate or speed up, and gather a team to work towards the goal. Redman and Hoerl suggest that your team take a "disciplined approach" - don't skip steps because the dataset is small for now! They also recommend providing relevant and hands-on training to speed up skill development.

    "Start small" is a resounding instruction, appearing in many guides to preparing your business for big data  and AI . As Redman and Hoerl so nicely put it, it "build[s] organisational muscle" and fosters data literacy. There's much to learn from the data you already have, and with a small dataset you can get started right away.

    If you start your data science projects while your data is small, your big data projects will be more likely to succeed, and you will have benefited from all the insights you gained along the way.

    Read Redman and Hoerl's full article here 

    1 Redman, T., Hoerl, R. (2019) Most analytics projects don't need much data, Harvard Business Review Magazine (July-August 2019 issue), available at  [Accessed 18 October 2019]
    2 Tarafdar, M., Beath, C.M., and Ross, J. W. (2019) Using AI to Enhance Business, MIT Sloan Management Review Magazine (Summer 2019), available at  [Accessed 17 October 2019]

    Topics: Data Analytics, AI adoption, Data culture, AI-Readiness

    Mia Hatton

    Written by Mia Hatton