Fusion Theme Blog | Strategic Data Management

Level up your data management

May 31, 2022

On the journey from data to analysis to insight, companies are shifting from a traditional approach and leaping forward into new ways of delivering actionable business intelligence. While the core goals remain the same — enabling data-driven decisions, optimizing cost efficiencies, and driving revenue growth — new tactics demand new skills. The pivot to a use-case model and good governance throughout the data lifecycle meets those challenges while also delivering faster time to insight. Connecting the business to fit-for-purpose data The new data mindset is purpose-driven. Based on specific use-cases generated by the business, today’s data teams build, deploy, and configure purpose-built data assets that meet the organization’s needs fast. This streamlined process represents a significant shift from the status quo for traditional data teams, but the streamlined workflow pays off. To generate fit-for-purpose data, start here: Solicit use cases from the business Understand and analyze the characteristics and dynamics of the use case Assess your existing data portfolio and identify information that might meet the need Consider the appropriate technology to synthesize data sets and deliver actionable insights Establishing a data asset creation workflow pays off in efficiency and value for IT and the business units involved. Learn more about how to develop data as an asset >> Speeding time to insight Traditional data warehousing models impose a high cost for integrating disparate data sets. A legacy workflow might include: Amending data architecture Creating a semantic model Time-consuming extract, transform, and load (ETL) processes for all data sets involved Preparing the data Making the data available for analysis Today’s businesses don’t have that long to wait for insights. Modern data technologies like Hadoop make it possible to stage data in a platform for immediate access. To structure your data and technology architecture toward a use-case driven model that fosters speed to insight key considerations include: A prioritized list of problems that need solutions Any characteristics or constraints that might impact time to value Available data assets and technologies, like data virtualization, that would enable you to access and analyze data in place Once you adopt methods for analyzing data in place, your team can deliver value on a much shorter timeline. Learn more about data architecture and integration >> Improving data literacy The demand for data-driven insights continues to accelerate. Companies at the forefront of the shift from volume to velocity use analytics pervasively throughout their organization and have the technology and agility to act on insights quickly. To become a competitive, speed-driven organization, your business must excel throughout the analytics lifecycle: Acquire: Harvest data quickly by exploring evolving big data technologies and optimizing first-party data strategies Analyze: Identify the most impactful insights Act: Implement the insights iteratively and strategically That final step involves your organization’s data literacy. Providing insights is one thing, but training your people to take the next right action on the data they see might require new skills. Upskilling the workforce to better understand and use data pays off richly in transformative accuracy, speed, and confidence. Learn more about building data literacy within your organization >> Implementing harvest-to-delivery data governance As the volume of available data continues to increase, businesses are building complementary abilities to understand and use it. But implementing tools and technology to harvest, integrate, and analyze data without robust governance frameworks opens companies up to significant risk. Building strong governance into your data asset creation and management workflows from the start can help. Learn more about how to implement good data governance >> Elevating data leaders As the world becomes more digital, and more customer behaviors move to a mobile context, businesses are changing to meet and match digital footprints with geospatial dimensions. Leadership must keep pace The need for a Chief Data Officer (CDO) at the table isn’t really a question anymore. Today, leading companies are asking where analytics and digital belong in the leadership playbook. To get the most value out of your data management, the right team members — with the right support and authority in place — could not be more important. Learn more about the importance of empowering your CDO >> Data management can be complex. A strategic viewpoint can help. Find out more about Fusion’s approach to strategic data management, or ask us your questions. Wherever you are on your data journey, we can help you keep moving forward.

How to choose a CDP

May 13, 2022

Organizations collect data from a wide range of sources and store it in any number of solutions, which can be spread across the business. For marketing departments trying to deliver personalized customer experiences, those silos present a problem, and a customer data platform (CDP) can seem like an easy answer. That could be the right conclusion, but it also might be premature. If you’re wondering how to choose a CDP, it makes sense to start with the basics. How to choose a CDP: 1. Understand the benefits and limitations of CDP solutions 2. Get internal alignment around the need and timing for a CDP 3. Surface your requirements, and prioritize your top 3-5 4. Map your core requirements to functionality you need in a CDP 5. Get stakeholder buy-in and secure a budget 6. Create a CDP selection matrix 7. Get external help if you need it 1. Understand the benefits and limitations of CDP solutions The CDP Institute defines a customer data platform as “packaged software that creates a persistent, unified customer database that is accessible to other systems.” This simple definition goes a long way toward unpacking the pros and cons of CDPs. A CDP is packaged software. PRO: A CDP is prebuilt and doesn’t require as much help from IT to implement and maintain as other solutions like custom platforms and data warehouses. CON: Because it’s packaged software and not a platform, a CDP will be less customizable and less efficient than adding customized functionality to an existing architecture. A CDP must create a persistent, unified customer database. PRO: A CDP ingests data from multiple sources across the business, standardizes it, and uses it to build an identity graph to enable real-time customer identification that fuels targeted multi-channel marketing efforts. CON: A CDP overcomes some of the difficulty of identity resolution but obscures the methodology. You get a unified customer view, but not a lot of certainty that it’s accurate. A CDP must be accessible to other systems. PRO: You can use the aggregated data and outputs from a CDP with your downstream systems. CON: The CDP you select must either come with the API connections you need for the rest of your martech and enterprise tech stacks or you’ll need to factor in costs for custom APIs. 2. Get internal alignment around the need and timing for a CDP Today’s martech stacks are growing fast, and it’s not always clear whether the functionality your marketing team needs already exists in the broader enterprise architecture. Overlapping features and blindspots are a problem — but also an opportunity. Gathering a cross-functional group that includes marketing, IT, data, and product teams can help you figure out how to get the most from your technology investments across the business. If you’re wondering how to choose a CDP, you might be surprised to find that many of the core functions you’re looking for in a CDP are already available in your CRM, MDM, data warehouse, analytics, and BI solutions. For example: Some analytical CRMs can track real-time online events like website browsing, adding to cart, and the like, much like CDPs. Your data warehouse may allow for an identity graph overlay and machine learning algorithms that can play a key role in enterprise-wide customer identity resolution. Your IT and data teams may already have identity and access management processes in place, and that single source of customer data truth could be integrated with your existing martech stack. So, how do you know if your organization needs a CDP? Your cross-functional group can explore potential use cases in light of departmental needs, budget, and existing functionality. While every organization is different, here are some ways to frame the conversation. You might need a CDP if: Your organization has a large volume of customer data stored in multiple places, and you either can’t or haven’t been able to integrate it into a single, real-time view Your marketing team can’t access customer data or perform data tasks without help from data and IT teams You can’t unify your online metrics, CRM data, and offline touchpoint and transactional data, making it hard to build a 360-degree view of your customer You’re moving to a first-party data strategy, but you don’t have systems in place to use your data to inform audience segmentation and personalized campaigns You have functionality gaps in your current martech stack that match up to CDP features You might NOT need a CDP if: You have a minimal and well architected martech stack Your customer data is simple or straightforward enough to analyze easily without additional tools Your marketing plan doesn’t require a lot of personalization, either because your products and services don’t require it, or because your roadmap doesn’t call for it in the short term Your customer data strategy has already mapped your needs to existing solutions and your roadmap doesn’t include a CDP Your budget doesn’t allow for duplicating storage costs, building and operating data ingestion processes, or keeping up with the steep total cost of CDP ownership Your security requirements don’t allow for third-party customer data storage Your internal IT and data teams find more affordable and secure means to implement identity and access management, democratize data access, and connect data storage with martech tools 3. Surface requirements, and prioritize your top 3-5 As you meet with your cross-functional team and discuss your need for a CDP, you can also surface requirements and use cases for a CDP solution. From your larger list, choose 3-5 top priorities to help you choose a CDP. Some common examples of CDP use cases include: Streamlining identity resolution, and making those outputs more accessible and actionable Combining online and offline data Creating more personalized content experiences on your website Using more strategic targeting in your multichannel campaigns Integrating and standardizing data across systems and making it easier to use those outputs in omnichannel marketing efforts 4. Map your core requirements to features you need in a CDP After you identify your top 3-5 use cases, map those core requirements to features you need in a CDP. For example, if one of your core requirements is enabling more targeted multichannel marketing campaigns in the EU, you might need to look for a CDP that offers GDPR-compliant identity resolution processes. Be sure to note which systems and solutions you’ll need to connect to your CDP both in terms of data ingestion and output sharing with downstream systems. Your CDP will need integrations and APIs to enable those connections. Common CDP integrations include CRMs, analytics tools and dashboards, advertising platforms, BI tools, data warehouses, and data lakes. Meet with your cross-functional group again to confirm your conclusions and assumptions. This meeting is a good time to clarify any security and data governance implications for your CDP selection as well. 5. Get stakeholder buy-in and secure a budget Your next step is to get stakeholders on board for the CDP acquisition and to establish funding to make the purchase. Leadership needs to know the full report from your cross-functional team, but they’ll also have questions about what kind of return the organization can expect from the investment As you frame up the value story, consider thinking about what not having a CDP costs your company: Is your marketing team using a significant amount of IT or data team hours looking for information? If you were to solve your data connectivity issues with custom-built APIs, how much would that cost? How much are you losing in customer value by not providing a customized experience? How much time is your marketing team spending manually moving data from one system to another? 6. Create a CDP selection matrix The vendor selection process doesn’t have to be painful. Your team can feel more confident about choosing a CDP because you’re going into the process with defined use cases and requirements, as well as support from the organization. That said, with an ever-growing CDP vendor landscape, it helps to have a process for narrowing down your choices. To accomplish that task, many companies use an internally framed selection matrix or decision tree, customized to fit CDP-specific needs and requirements. These tools can help you think through solution options using your own criteria. However you handle the vendor review, plan to bring a short list of possibilities back to your cross-functional team for a final discussion before making a purchasing decision. 7. Get external help if you need it. Figuring out how to choose a CDP and where the solution fits into your larger customer data strategy isn’t easy. Fusion Alliance helps companies navigate changing customer data environments with a unique methodology that fosters collaboration, transparency, and shared ownership of digital transformations.   Whether you’re just getting started or stuck in the messy middle of a CDP selection process, we help you unpack your processes and partnerships to identify risks and opportunities so you can take the right next steps toward a future-focused customer data strategy.   Let’s talk. Keep reading: The Ultimate Guide to Creating Your Customer Data Strategy

CDP vs CRM: 5 key questions to inform your decision

Apr 22, 2022

The difference between a customer data platform (CDP) and customer relationship management (CRM) solution may be difficult to determine at first, because both options collect, store, and put customer data to use in support of business goals. While their functions may overlap, the CDP vs CRM debate becomes easier when you get clarity about the people, processes, and use cases for each option. How to make the CRM vs CDP decision 1. What is a CDP? 2. What is a CRM? 3. What data is collected by a CRM vs CDP? 4. Who uses a CDP vs CRM and for what purpose? 5. What do we need: a CDP, CRM, or both? 1. What is a CDP? A CDP unifies and standardizes large and detailed data sets from a wide variety of sources, resulting in robust customer profiles that enable real-time personalization. The CDP Institute defines a CDP as “packaged software that creates a persistent, unified customer database that is accessible to other systems.” Additionally, a CDP must have the following capabilities: Ingest data from any source Capture full detail of ingested data Store ingested data indefinitely (subject to privacy constraints) Create unified profiles of identified individuals Share data with any system that needs it Through the process of identity resolution, the CDP can match, merge, and deduplicate data into a single customer view that can be segmented and analyzed — by human analysts or with the assistance of machine learning. 2. What is a CRM? A CRM and a CDP are both software solutions that handle customer data, but they differ in how, why, and who for. The difference came about organically, as organizations adopted different use cases for their customer data over time. “CRM solutions were often proposed to tackle customer data management problems. The idea was that you could get ‘all of your data in one place’ to use for sales, marketing, and customer service. The promise was they’d break down silos in enterprises and design a view of the customer that wasn’t specific to sales or marketing or customer service. That sounds familiar to the promise of CDPs, doesn’t it?” — Lizzy Foo Kune, senior director analyst at Gartner A CRM helps organizations manage customer relationships by consolidating what is known about customers from one-to-one touchpoints and transactional details into a single database, giving sales and service teams personal and actionable insights. According to the Microsoft Dynamics 365 website “CRM systems help you manage and maintain customer relationships, track sales leads, marketing, and pipeline, and deliver actionable data.” Sound similar to a CDP? There’s a key difference: CRMs only apply to known customers and contacts. Moreover, they don’t cleanse, combine, standardize, or deduplicate the customer records, so they can’t give a business a “single customer view” across channels. 3. What data is collected in a CRM vs CDP? That key difference reflects the two business silos that CRMs were developed to unite: marketing and sales. Marketing needs a high volume of customer data across touchpoints in a single, unified view to understand your customers and their behavior. CDPs collect digital data automatically using integrations and code snippets embedded in digital touchpoints, gathering customer data from websites, laptops, mobile devices, apps, and even CRMs into one place. The CDP then cleans it, and produces consolidated customer. Sales needs customer data to help manage the customer relationship. CRMs store historical data about customer interactions in order to inform future interactions. The data CRMs collect is usually entered manually and its purpose is tightly focused on logging an interpersonal or transactional interaction — for example, notes from the latest sales call – to inform future interactions. The data inputs are simple, although difficult to standardize or automate, and are usually done manually by sales (and service) people to track the progress of the relationship. 4. Who uses a CDP vs CRM and for what purpose? Your organization’s CDP vs CRM discussions may come down to who needs to use the system to accomplish critical business tasks. As we’ve said above, marketers need a unified view of the customer’s entire experience of the brand over time. A CDP’s ability to ingest, cleanse, manage, and analyze large volumes of data from many digital sources makes that task easier. But for sales and support teams, the key driver is managing customer relationships. In these customer-facing roles, contact management is critical, so a CRM’s ability to capture notes and manual inputs about one-to-one interactions facilitates that function. 5. What do we need: a CRM, CDP, or both? While choosing between solutions isn’t easy, it’s not necessarily an either/or decision. You might find a both/and solution serves your business better. How do you make the call? If your business primarily needs to manage customer relationships in a more detailed, efficient, and personalized way, you might choose a CRM. In fact, over the last few years, CRMs have been innovating and evolving to function more and more like CDPs so it might be prudent to wait and/or choose vendors carefully. Gartner predicts that 70% of independent CDP vendors will be acquired by larger technology vendors or will diversify by 2023.  “CRM systems have seen the competitive threat that CDPs brought to the table,” Gartner’s Foo Kune said. “As CRM technologies recognize that they need to update their aging databases to meet the needs of modern business functions, including marketing, augmenting your CRM with a CDP may be unnecessary.” If your business primarily needs to have a broad view of who your customers are and how they engage with your business, you may opt for a CDP. “Companies seeking a new strategy to form personalized customer experiences through data will need a CDP as it offers the resources to create a comprehensive view of the customer across each platform they interact with in real-time — whether it’s social media, apps or mobile,” says Heidi Bullock of Tealium, a CDP provider. “CRMs, on the other hand, help manage sales-focused customer data rather than collecting data across different channels.” And, if your business is broad you can choose both a CDP and a CRM.  While CDPs and CRMs offer two different marketing and sales data management solutions with differing strengths, you don’t necessarily have to choose between them. “CDPs and CRMs can actually operate simultaneously, as they work to fulfill different business goals,” Tealium’s Bullock notes. It’s possible to use a CRM as an input and output channel to a CDP, and, in turn, use a CDP to provide a 360° customer view data set within the CRM. Choosing both a CDP and a CRM can deliver both an amazing customer experience and tremendous business value: achieving high marks in customer satisfaction and providing integrated tracking and engagement. The CDP vs CRM choice depends on your roadmap. Fusion works with clients to define a customer data strategy that fits each organization’s unique strategic objectives, operational needs, and timeline. From there, our team creates a tactical roadmap to define actionable steps toward those goals. Whether you’re just getting started or trying to get your digital transformation back on track, we can help. Ask a question >> Book a workshop >> Learn more about customer data strategy >>

Data executives empowered: The importance of the data community and elevating the CDO

Jul 22, 2021

Almost 20 years ago, Capital One recognized the need for one person to oversee their data security, quality, and privacy, and the role of the Chief Data Officer was born. Now reports show that 68% of organizations have a CDO (Harvard Business Review, 2020). And while the role has become more common and has significantly evolved, many data executives are still struggling to get a seat at the table or bring data to the forefront of their organization. In fact, in a recent survey, only 28% of respondents agreed that the role was successful and established. Company leaders agree that there needs to be a single point of accountability to manage the various dimensions of data inside and outside of the enterprise, including the quality and availability of that data. But now we are at a crossroads — what is the best way to align the work that the CDO does with the strategy of the business as a whole? The reality is that CDOs often struggle to find the internal and external support and resources needed to educate others to align with the organization’s goals. Implementing enterprise data governance, data architecture, data asset development, data science, and advanced analytics capabilities — such as machine learning and video analytics — at scale, is not an easy task. To be successful, data executives need support, resources, and communities focused on the elevation of data. We are proud to continue to help these communities come to life for the benefit of our colleagues and clients, establishing local resources here in the Midwest with global scale, reach, and impact. Read on as Mark Johnson, our Executive Leader for Data Management and Analytics, provides insight on the current state of data and the CDO, and provides details on multiple opportunities for data leaders of different levels to get more involved in the data community. Q: How has the role of data changed/evolved for organizations? The reality is that information is everything. This global pandemic proved that to many organizations. For some, it showed that their digital network was ready, and they were aptly prepared to take on COVID. For others, it has forced them to recognize their own immaturity with data and analytics. On its own, managing data is not exciting — the information just sort of exists. To give data value, you have to put it to use. And so, I think we are going to see the Chief Data Officer and/or Chief Data Analytics Officer really find their own in the coming years. It’s time for their seat at the table. The C-suite is now asking questions that can only be answered with data, and now they truly understand both the value and consequences of the data game. Q: What do you think are the biggest challenges facing CDOs/data leaders today? I think that the biggest challenge for data executives today is the acquisition of talent that is seasoned and experienced where you need them to be for your organization. Higher education hasn’t necessarily kept up with the data world, and often times it takes additional training to reach the right levels. The reality is that right now the talent is manufactured in the real world. Data executives have to be connected and equipped to mentor, train, and keep the right people. Q: You’ve mentioned that data leaders need to connect with each other. What value can people expect from these data communities? I think there is tremendous value. As we are seeing the power of data evolve in organizations, and the role of data leaders evolve as well, I think coming together to collaborate and share elevates the leader, the organization, and the view of data as a whole. In these communities, it gives people a safe space to talk about how they are doing, what they are doing, what their biggest challenges are, and what solutions are working for them. These communities have truly become both a learning laboratory and an accelerator for data. Q: As a big proponent of connecting data leaders, you have been involved in creating different opportunities for people to get together. What groups/events would you recommend, and how can people get involved? I personally have been involved with the MIT Chief Data Officer and Information Quality Symposium (MIT CDOIQ), which is such a great opportunity to start with for connection. It has developed into additional opportunities for data leaders at all levels to get involved and create the kind of community we need to truly elevate the value of data. Organizations like the CDO Magazine, the creation of CDO roundtables across the nation, and the International Society of Chief Data Officers (isCDO) all evolved from connecting data leaders and identifying common challenges. MIT CDOIQ: The International MIT Chief Data Officer and Information Quality Symposium (MIT CDOIQ) is one of the key events for sharing and exchanging cutting-edge ideas and creating a space for discussion between data executives across industries. While resolving data issues at the Department of Defense, the symposium founder, Dr. Wang, recognized the need to bring data people together. Now in its 15th year, MIT CDOIQ is a premier event designed to advance knowledge, accelerate the adoption of the role of the Chief Data Officer, and change how data is leveraged in organizations across industries and geographies. Fusion has been a sponsor of this symposium for seven years now, and we are so excited to see how the event has grown. Designed for the CDO or top data executive in your organization, this is a space to really connect with other top industry leaders. CDO Roundtables Fusion has always been focused on building community and connecting people. And when one of our clients, a Fortune 500 retailer, mentioned wanting to talk with other data leaders from similar corporations, we realized that there was a big gap here — there was no space that existed where data leaders could informally come together, without sales pitches and vendor influence, and simply talk. That’s how the CDO roundtables were born — a place that allows data leaders to get to know each other, collaborate, accelerate knowledge growth, and problem solve. We just started two years ago in Cincinnati, but now, now we’ve expanded to multiple markets including Indianapolis, Columbus, Cleveland, Chicago, and Miami. These groups are designed for your CDO/CDAO and truly create an environment for unfiltered peer-to-peer discussion that helps solves data leadership challenges across industries. If you’re interested in joining one of these roundtables or starting one in your market, email me or message me on LinkedIn. I’m here and ready to get these roundtables started with executives in as many communities as I can. The more communities we have, the more data leaders and organizations we can serve. International Society of Chief Data Officers (isCDO) Launched out of the MIT CDOIQ symposium, the isCDO is a vendor-neutral organization designed to promote data leadership. I am excited to be a founding member of this organization, along with our Vice President of Strategy, David Levine. Our ultimate goal is to create a space that serves as a peer-advisory resource and enables enterprises to truly realize the value of data-driven decision making. With multiple membership options available, isCDO is the perfect opportunity for data leaders looking to connect with their peers and gain a competitive advantage by focusing on high-quality data and analytics. CDO Magazine I am really proud to be a founder of the CDO magazine, as it really is a resource for all business leaders, not just the CDO. We designed the magazine to be a resource for C-suite leaders — to educate and inform on the value proposition, strategies, and best practices that optimize long-term business value from investments in enterprise data management and analytics capabilities. Check out the publication here. And if you’re interested in contributing content or being interviewed, let me know at mjohnson@fusionalliance.com. Closing: The role of the CDO is integral to organizations, but it’s still evolving. Now more than ever, it is important that data leaders come together to collaborate and problem-solve. Fusion is excited to be a part of each of these initiatives, and we are committed to being an agent of change in the communities we serve and beyond. By connecting global thought leaders we believe that organizations will realize the value of data to power their digital transformation. If you’re interested in joining any of these data communities or just have questions, feel free to reach out to Mark via email or on LinkedIn.

The rise of big data technologies and why it matters

Mar 24, 2021

Getting the right data to the right people at the right time is the name of the game in today’s demanding marketplace. Every company has to find a way to harness big data and use it to drive growth. And if your organization isn’t talking big data, you are at a competitive disadvantage. This article covers a top-level view of big data’s evolution and key components. It can help you understand the importance of big data and technologies that are essential to its discussion. With this foundation, you can proceed to the next step — addressing what to do with your data and how. Just how much data exists? Every passing moment, the pace of data creation continues to compound. In the time it takes you to read these few paragraphs there will be: more than 200 million emails sent millions of dollars in e-commerce transacted 50 hours of YouTube videos uploaded millions of Google searches launched tens of millions of photos shared Every few minutes, this cycle repeats and grows. In 2019, 90% of the world’s digital data had been created in the prior two years alone. By 2025, the global datasphere will grow to 175 zettabytes (up from 45 zettabytes in 2019). And nearly 30% of the world’s data will need real-time processing. Over the last decade, an entire ecosystem of technologies has emerged to meet the business demand for processing an unprecedented amount of consumer data. What is big data? Big data happens when there is more input than can be processed using current data management systems. The arrival of smartphones and tablets was the tipping point that led to big data. With the internet as the catalyst, data creation exploded with the ability to have music, documents, books, movies, conversations, images, text messages, announcements, and alerts readily accessible. Digital channels (websites, applications, social media) exist to entertain, inform, and add convenience to our lives. But their role goes beyond the consumer audience — accumulating invaluable data to inform business strategies. Digital technology that logs, aggregates, and integrates with open data sources enables organizations to get the most out of their data, and methodically improves bottom lines. Big data can be categorized into structured, unstructured, and semi-structured formats. The development of modern data architecture Until recently, businesses relied on basic technologies from select vendors. In the 1980s, Windows and the Mac OS debuted with integrated data management technology, and early versions of relational database engines began to become commercially viable. Then Linux came onto the scene in 1991, releasing a free operating system kernel. This paved the way for big data management. What is big data technology? Big data technologies refer to the software specifically designed to analyze, process, and extract information from complex data sets. There are different programs and systems that can do this. Distributed file systems In the early 2000s, Google proposed the Google file system, a technology for indexing and managing mounting data. A key tenet to the idea was using more low-cost machines to accomplish big tasks more efficiently and inexpensively than the hardware on a central server. Before the Information Age, data was transactional and structured. Today’s data is assorted and needs a file system that can ingest and sort massive influxes of unstructured data. Open-source and commercial software tools automate the necessary actions to enable the new varieties of data, and its attendant metadata, to be readily available for analysis. Hadoop Inspired by the promise of distributing the processing load for the increasing volumes of data, Doug Cutting and Mike Cafarella created Hadoop in 2005. The Apache Software Foundation took the value of data to the next level with the release of Hadoop in Dec. 2011. Today, this open-source software technology is packaged with services and support from new vendors to manage companies’ most valuable asset: data. The Hadoop architecture relies on distributing workloads across numerous low-cost commodity servers. Each of these “pizza boxes” (so called because they are an inch high and less than 20 inches wide and deep) has a CPU, memory, and disk storage. They are simple servers with the ability to process immense amounts of various, unstructured data when running as nodes in a Hadoop cluster. A more powerful machine called the “name node” manages the distribution of incoming data across the nodes. By default, data is written to at least three nodes and might not exist in its entirety as a single file in any one node. Below is a simple diagram that illustrates the Hadoop architecture at work. Open source software The majority of enterprises today use open source software (OSS). From operating systems to utilities to data management software, OSS has become the standard fare for corporate software development groups. Serving as a progressive OSS organization, Apache Software Foundation is a non-profit group of thousands of volunteers who contribute their time and skills to building useful software tools. As the creators, Apache continuously works to enhance Hadoop code — including its distributed file system called Hadoop Distributed File System (HDFS) — as well as the code distribution and execution features known as MapReduce. Within the past few years, Apache released nearly 50 related software systems and components for the Hadoop ecosystem. Several of these systems have counterparts in the commercial software industry. Vendors have packaged Apache’s Hadoop with user interfaces and extensions, while offering enterprise-class support for a service fee. In this segment of the OSS industry, Cloudera, Hortonworks, and Pivotal are leading firms serving big data environments. Now software systems are so tightly developed to the core Hadoop environment that no commercial vendor has attempted to assimilate the functionality. The range of OSS systems, tools, products, and extensions to Hadoop include capabilities to import, query, secure, schedule, manage, and analyze data from various sources. Storage Corporate NAS and SAN technologies, cloud storage, and on-demand programmatic requests returning JSON, XML, or other structures are often secure repositories of ancillary data. The same applies to public datasets — freely available datasets, in many cases for economic activity by industry classification, weather, demographics, location data, and thousands more topics. Data of this measure demands storage. Distributed file systems greatly reduce storage costs while providing redundancy and high availability. Each node has its local storage. These drives don’t require speed or solid-state drives, commonly called SSDs. They are inexpensive, high-capacity pedestrian drives. Upon ingestion, each file is written to three drives by default. Hadoop’s management tools and the Name Node monitor each node’s activity and health so that poorly performing nodes can be bypassed or taken out of the distributed file system index for maintenance. The term “data lake” describes the vast storage of different types of data. These vastly different data sources are in a minimum of a dozen different file formats. Some are compressed or zipped. Some have associated machine data, as found in photos taken with any phone or digital camera. The date, camera settings, and often the location are available for analysis. For example, a query to the lake for text messages that included an image taken between 9 p.m. and 2 a.m. on Friday or Saturday nights in Orlando on an iPhone would probably show fireworks at Disney World in at least 25% of the images. Administration of big data initiatives The enterprise administration of applications — their storage requirements, security granularity, compliance, and dependencies — required Hadoop distributions to mature these capabilities in the course of becoming a managed service to an enterprise (like those from Cloudera and Hortonworks). In the graphic above, you can see a view of Hadoop’s place among other software ecosystems. Note that popular analysis tools (below) are valuable in developing big data solutions: Excel and Tableau databases such as SQL Server and Oracle development platforms such as Java and Informatica Data Quality Administration through Cisco and HP tools is common. Further development of big data Commercial software companies have begun connecting to Hadoop, offering functionality such as: data integration quality assessment context management visualization and analysis from companies such as IBM, Microsoft, Informatica, SAP, Tableau, Experian, and other standard carriers. Analytics and data science Analytics is the endgame for developing a big data environment. The rise of big data has given credence to a new resource classification, the data scientist — a person who embodies an analyst, technologist, and statistician all in one. Using several approaches, a data scientist might perform exploratory queries using Spark or Impala, or might use a programming language such as R or Python. As a free language, R is rapidly growing in popularity. It is approachable by anyone who is comfortable with macro languages such as those found in Excel. R and its libraries implement statistical and graphical techniques. Moving to the cloud Cloud computing is very different from server-class hardware and software. It involves cloud storage, multi-tenant shared hosts, and managed virtual servers that are not housed on a company’s premises. In cloud environments, an organization does not own equipment, nor does it employ the network and security technologists to manage the systems. Cloud computing provides a hosted experience, where services are fully remote and accessed with a browser. The investment to build a 10- or 20-node HDFS cluster in the cloud is relatively small compared to the cost of implementing a large-scale server cluster with conventional technologies. The initial build-out of redundant centers by Amazon, Microsoft, Google, IBM, Rackspace, and others has passed. We now have systems available at prices below the cost of a single technician. Today, cloud computing fees change rapidly with pricing measured by various usage patterns. Conclusion: The rise of big data is evident Big data is not a fad or soon-to-fade trending hashtag. Since you began reading this article, more than 100 million photos have been created, with a sizeable portion having a first-degree relationship to your industry. And the pace of data creation continues to increase. The distribution of computing processes can help organizations to gain a 360-degree view of their customers through big data collection and analysis. And companies that embrace big data technologies and solutions will rise ahead of their competitors. Big data technologies are becoming an industry standard in finance, commerce, insurance, healthcare, and distribution. Welcoming big data technologies and solutions is key in the optimization and continued growth going forward. Companies that embrace data solutions can continue to improve management and operational processes and create a competitive advantage to withstand an ever-evolving marketplace.

A winning data management and analytics strategy

Sep 8, 2020

Digital natives like Uber and Lyft have all but changed the face of the taxi industry and the customer experience. Many are huge fans of these companies, and it’s no wonder. You don’t have to flag anyone down or have an awkward street scuffle to get a ride. Uber has skipped the web and gone straight to the mobile device as its target platform for order management and fulfillment. Every customer transaction goes into Uber’s database: name, address, credit card info, cell number, pickup location, drop-off location, where you travel and when — the list goes on. But it’s not just the ride that is valuable to Uber. Their database is where the real value is. And guess what? Uber was recently valued at $49 billion. Every company has data. And every company needs a data strategy to take advantage of its value. Some steps to create a winning strategy are: 1. Make data management and analytics a priority Increasingly, business and personal transactions and interactions are going through digital channels. As they move to digital channels, they leave behind a lot of data that was not available to companies before. New sources of data exist everywhere — social media, geolocation data, etc. There’s just a lot more digital content available now to help a business understand its performance, relationships, and reputation. But you need to know what to do with it. For example, as Uber’s database grows, it becomes more valuable. They can look at their customers’ digital footprint through analytics and, over time, they can see distinct types of users emerge from their travel and interaction patterns. They can then use these analytics to expand and refine their service offering to better serve the needs of users and travelers with similar digital footprints. 2. Overcome IT challenges There are many new choices of data technologies. You need to figure out how to incorporate them into your company’s existing technology stack. But even before that, you need to understand how to manage data as an asset and consider: How data is governed Who owns the data How to manage data quality and security How to handle demand management as new data requests come in and new data sources are identified In addition, you need to understand how it will be integrated into the infrastructure and environment. That said, it’s hard to find the people right now who understand all of these technologies. There is a shortage of data scientists and Hadoop engineers, for example. Having access to the resources with the skills to implement and manage these new technologies can be one of your biggest constraints and barriers. Legacy systems vs. open source There are also challenges associated with the whole technology space. Legacy vendors, such as Oracle, Teradata, and Microsoft, want to maintain their hold. They’re all fighting to remain relevant in a technology market space where open source is creating more compelling and cost-effective solutions for businesses. Microsoft, which has a huge research component, has had difficulty embracing the open source movement in the past. Today they are in full support of open source projects in Azure and Visual Studio and release many of their own code bases, such as .NET. Open source is actually much more valuable because it’s run by people who are constantly working on issues of security or lack of performance. These folks will immediately address your issues for one overriding reason, they are passionate about code — Wikipedia all over again! 3. Prepare for organizational change We’re not just dealing with our own transactional data anymore. We’re dealing with data from our industry or sector, as well as external data, such as weather. Though weather might not seem like it has anything to do with your business, weather data can provide great insight that can enable you to positively impact business. Many other datasets are also available, some for a fee, which organizations have discovered they must pay attention to, in addition to their own operational data. The technologies of today’s data management agenda are new and emerging and are not technologies for which IT traditionally has the skills. There’s a big divide between IT capability and what the business demands for integrating and managing data. As a result, roles like data scientist and data analyst, with the kinds of necessary skills, are not yet common within organizations, making organizational and change management a requirement. 4. Embrace the role of a CDO Until recently, we’ve pretended as if the people who are responsible for the technology (the wires, pliers, software, and ERP systems) actually care about the data, but they don’t. A CIO is not the best person to manage the data. There’s a new paradigm out there, the chief data officer or CDO. Data is such a critical corporate asset that it needs to be managed strategically and at the executive level outside of IT. Technology is an enabler, but data is an asset. Currently many account for them in the exact opposite paradigm. Many organizations are now appointing a CDO, reporting to the COO or CEO, and their role is to oversee and manage quality, integrity, and use of the organization’s data assets, just like the CFO governs the organization’s financial data. Start implementing a winning data strategy today The elements covered here will get your business off to a strategic start toward more effective management of your data and analytics. If you want to be successful, remain open to new ideas, get help from outside, and embrace new paradigms for how your business should interact with data as it continues to evolve. It’s critical to understand that you should absolutely take advantage of anything that can accelerate driving insights from the data already just sitting in your systems out into the marketplace. Having a strategic partner who brings the required expertise and ability to implement proven methodologies will enable your company to create successful data capabilities, and you’ll be able to groom and train internal resources at the same time. That’s what’s going to enable you to beat your competition.