Over the past year, I've encountered frequent questions from executives in Thailand, including:

1. “We have a lot of data, but we don't know what to do with it.”

2. “I don't know how to start with big data or data analytics.”

3. “Everyone is starting to use big data, AI, machine learning. We need to start doing something too.”

4. “We've invested a lot in tools, but we're not sure how to maximize their benefits.”

5. “Are there any use cases where we can apply technology right away? We want something that is ready to go.”

This led me to want to share lessons learned with various organizations that encounter the traps of big data that should be avoided, in order to prepare for the implementation of data analytics in organizations. Based on my experience working with both public and private sectors, both domestically and internationally (I may use big data and data analytics interchangeably, but personally, I believe data analytics is more appropriate, as when people talk about big data, they inevitably have to use data analytics to create value).

Lessons Learned

1. Organizations Need a Clear Data Analytics Strategy

Successful organizations should have an attitude of transforming their corporate culture to establish a clear strategy for data utilization (strategic data-driven organization), such as:

- Having a decision-making framework on how to use data, determining at which operational steps data should be utilized, so that departments can be trained and benefit maximally from data.

- Managing various departments within the organization (organizational management) to facilitate data sharing for the collective benefit of the organization or lessons learned from data utilization.

- Having clear support from executives to drive the use of data analytics to permeate every level of the organization, creating an analytics culture. For example, Amazon and Netflix utilize analytics in nearly every aspect from technology to business.

When I worked for a consulting firm that served a U.S. government agency, I had the opportunity to observe the creation of analytics capabilities in these two organizations (both private and public) simultaneously. I can say that the executives of both organizations understood why they needed to use data analytics and supported the establishment of a central unit reporting directly to executives, with a team of data scientists managing analytics work, disseminating knowledge, and collaborating with various business units, as they viewed the application of data analytics as a competitive advantage necessary for increasing revenue and enhancing operational efficiency.

Upon returning to Thailand, I encountered many organizations that often expressed similar sentiments: the senior management recognizes the importance of big data (which has become a prominent trend this year) and has budgeted for various projects. However, when I inquired about their strategy for utilizing big data, some individuals became silent and could not respond, as there was a lack of understanding of whether the vision of the “senior management” aligned with the thoughts of the operational staff. Therefore, senior management may need to clarify the strategy with the operational staff more explicitly.

2. The Illusion of Big Data

Organizations often fall into the illusion that having a large amount of stored data is sufficient, and they seek tools to manage and derive benefits from that data. These statements are often created by various vendors trying to entice organizations to purchase tools or infrastructure. However, the essence of big data does not solely depend on the volume of data but also on the following factors:

- Data Quality

High-quality data is data that is in a state suitable for addressing business needs. For example, the data used must not have excessive missing values, must have multiple variables, and the data within each variable must exhibit variability, meaning they should not all look the same.

In predictive modeling, such as fraud detection models, clear labels are needed to differentiate data, indicating which data points represent fraudulent behavior and which do not.

- Data Accessibility and Integration (single version of the truth)
Many organizations face challenges because their internal data is fragmented and may be stored separately by various departments, creating obstacles to aggregating data for maximum benefit. For instance, a bank may want to analyze customer data comprehensively along the customer journey but cannot do so due to missing crucial information from retail or credit card departments.

- Utilizing Under-Utilized Data Sets

Data is valuable regardless of its format, yet approximately 80-90% of data is often in the form of unstructured data (images, videos, audio, text) or dark data (call/server logs, emails, mobile geolocation, machine data), which are often not analyzed due to difficulties in organizing and structuring the data. However, these data should be collected and processed to derive greater benefits.

3. Assuming Technologists or Computer Engineers Can Create Ready-Made Analytics Programs

Most organizations, especially IT departments, tend to overestimate the capabilities of technology, believing that it can immediately perform analytics tasks. This misconception often arises when purchasing decisions rely solely on the IT department without input from business users regarding the actual purpose of the analytics tools, who will use them, and whether those users possess sufficient skills and expertise.

Analytics work differs from IT or software development, as users of analytics tools must understand data, possess analytical questioning skills, and comprehend the limitations of various technologies and tools.

A common issue in both public and private organizations is the focus on purchasing tools without adequately training or developing personnel to use and maintain them. Some employees may not even understand the objectives behind acquiring these tools, leading to significant resource waste for the organization. The result is that users cannot maximize the benefits of these tools, and another issue arises: a lack of accountability in tracking whether the investment has benefited the organization or created new capabilities.

Another point encountered is that some organizations only consider purchasing technology or imitating successful case studies from others without realizing that the data and context of the problems faced by each organization differ, which may lead to different outcomes. Therefore, good executives should have the vision to support analytics projects that arise from effective collaboration between IT and business users, with accountability for measuring clear returns on investment, such as developing analytics skills within the organization and ensuring that purchased tools are beneficial.

4. The Timeline of Data Analytics Projects Differs from IT Projects

One of the risks of data analytics projects (especially for organizations that do not yet have a clear data analytics plan) is whether the available data is ready for use and whether data access is feasible. The data preparation process can take up to 60-90% of the project timeline, which is different from IT projects where hardware and software components are more certain.

I often hear project managers wanting detailed project specifications, which is often unrealistic due to the inherently flexible nature of projects, especially during the data preparation phase. Managing data analytics projects requires repeated iterations to check the readiness and quality of the data, using adjustments in the data as lessons learned for future data governance. From my experience, allowing a bit more time for this can significantly reduce risks.

5. Developing People Inside and Outside the Organization Is More Than Just Training Classes or Meetups

A major obstacle that both public and private organizations currently face is the lack of personnel with sufficient knowledge and skills to work in data analytics. Don't just think about finding individuals to become data scientists (those who can analyze and understand various statistical techniques or machine learning/AI algorithms), as even individuals with analytical skills are hard to come by. What can be done is to develop internal talent by providing opportunities for those with backgrounds in mathematics, statistics, or computer science/engineering to work alongside business units to solve real problems, rather than just sending them for training, as on-the-job training is often more effective.

New knowledge can emerge from collaboration and processes that facilitate knowledge exchange within teams, known as collective intelligence (see the collective intelligence column in 1,2,3 that I previously discussed). Additionally, bringing in external experts to help build and educate internal staff (train the trainer) will accelerate the development of data analytics.

Moreover, selecting experts may require verification or testing of their capabilities based on various factors such as knowledge, experience, and past work that could be applied to the organization's problems.

Furthermore, making analytics tools user-friendly for office users and educating internal personnel will raise awareness of data analysis tools and increase demand for analytics within the organization.

Over the past year, there have been numerous events in Thailand, including conferences, meetups, and workshops related to data analytics, big data, data science, and AI. However, most of these events have been informational or case study presentations. I believe it is time to change the purpose and elevate the content of these events to be more technical to foster a maker community (maker = doers).

6. Share Success Stories of Data Utilization with People in the Organization

Building data analytics capability within an organization is about changing the mindset of its people. Therefore, it is essential to start with quick-win projects that can create impact in a short time to build confidence among the team and the supporting executives, while also raising awareness of the benefits of data utilization, stimulating interest, and creating demand for data analytics projects.

From everything mentioned, the most crucial first step for organizations is to define business objectives: what do we want to achieve with data? It should not start with technology or the trend of big data. Instead, it may begin with studying data analytics case studies to see how other organizations have utilized data and how it has supported their business, such as increasing sales, reducing costs, enhancing employee efficiency, or improving customer experience. When we start with clear objectives, we can see the direction of work and the connections regarding what data to use or collect.

Another aspect that organizations often overlook is their existing advantages, which lie in the data already within their systems. Focusing solely on technology or relying on any one tool may yield opposite results, as modeling techniques change rapidly. Having exclusive rights to the IP of work does not create as much competitive advantage. There are other business models that can benefit from analytics capability, such as open-source with added features, revenue sharing, or joint investments with companies that excel in this area. Companies like Google, Facebook, Amazon, and Microsoft possess vast amounts of valuable data and skilled personnel capable of effectively utilizing that data. These companies allow the use of developed tools or techniques to create value from their data, fostering a workforce capable of meeting the market's demand for such talent and promoting competition that positively impacts the development of data analytics talent in the country in the long term.

There is much more I would like to share. Next time, I will discuss various forms of thinking to address organizational problems.

Thank you for the information from thaipublica.org