佛瑞斯特研究公司（Forrester）的研究人员发现，2016年，近40％的公司正在实施和扩展大数据技术应用，另有30％的公司计划在未来12个月内采用大数据技术。2016年NewVantage Partners的大数据管理调查发现，62.5％的公司现在至少有一个大数据项目投入生产，只有5.4％的公司没有大数据应用计划，或者是没有正在进行的大数据项目。

研究人员称，会有越来越多的公司加速采用大数据技术。互联网数据中心（IDC）预测，到2020年大数据和分析技术市场，将从今年的1301亿美元增加至2030亿美元。“公司对数据可用性要求的提高，新一代技术的出现与发展，以及数据驱动决策带来的文化转变，都继续刺激着市场对大数据和分析技术服务的需求”， IDC副总裁Dan Vesset表示。 “2015年该市场全球收入为1,220亿美元，预计到2016年，这一数字将增长11.3％，并预计在2020年以11.7％的复合年增长率（CAGR）继续增长。”

虽然大数据市场将会继续增长这一点毋庸置疑，但企业应该如何应用大数据呢？目前还没有一个清楚的答案。新的大数据技术正在进入市场，而一些旧技术的使用还在继续增长。本文涵盖大数据未来发展的十大趋势，这些趋势可能对2017年及以后的大数据市场产生极大影响。

专家预计，机器学习、预测分析、物联网和边缘计算将对2017年及以后的大数据项目产生深远影响。

Apache Hadoop、Spark等开源应用程序已经在大数据领域占据了主导地位。一项调查发现，预计到今年年底，近60％企业的Hadoop集群将投入生产。佛瑞斯特的研究显示，Hadoop的使用率正以每年32.9％的速度增长。

专家表示，2017年许多企业将继续扩大他们的Hadoop和NoSQL技术应用，并寻找方法来提高处理大数据的速度。

很多公司正试图加速大数据处理过程，它们采用的一项技术就是内存技术。在传统数据库中，数据存储在配备有硬盘驱动器或固态驱动器（SSD）的存储系统中。而现代内存技术将数据存储在RAM中，这样大大提高了数据存储的速度。佛瑞斯特研究的报告中预测，内存数据架构每年将增长29.2％。

目前，有很多企业提供内存数据库技术，最著名的有SAP、IBM和Pivotal。

随着大数据分析能力的不断提高，很多企业开始投资机器学习（ML）。机器学习是人工智能的一项分支，允许计算机在没有明确编码的情况下学习新事物。换句话说，就是分析大数据以得出结论。

高德纳咨询公司（Gartner）称，机器学习是2017年十大战略技术趋势之一。它指出，当今最先进的机器学习和人工智能系统正在超越传统的基于规则的算法，创建出能够理解、学习、预测、适应，甚至可以自主操作的系统。

预测分析与机器学习密切相关，事实上ML系统通常为预测分析软件提供动力。在早期大数据分析中，企业通过审查他们的数据来发现过去发生了什么，后来他们开始使用分析工具来调查这些事情发生的原因。预测分析则更进一步，使用大数据分析预测未来会发生什么。

普华永道（PwC）2016年调查显示，目前仅为29％的公司使用预测分析技术，这个数量并不多。同时，许多供应商最近都推出了预测分析工具。随着企业越来越意识到预测分析工具的强大功能，这一数字在未来几年可能会出现激增。

企业使用机器学习和AI技术的另一种方式是创建智能应用程序。这些应用程序采用大数据分析技术来分析用户过往的行为，为用户提供个性化的服务。推荐引擎就是一个大家非常熟悉的例子。

在2017年十大战略技术趋势列表中，高德纳公司把智能应用列在了第二位。高德纳公司副总裁大卫·希尔里（David Cearley）说：“未来10年，几乎每个app，每个应用程序和服务都将一定程度上应用AI。

许多企业也将大数据分析纳入安全战略。企业的安全日志数据提供了以往未遂的网络攻击信息，企业可以利用这些数据来预测并防止未来可能发生的攻击，以减少攻击造成的损失。一些公司正将其安全信息和事件管理软件（SIEM）与大数据平台（如Hadoop）结合起来。还有一些公司选择向能够提供大数据分析能力产品的公司求助。

物联网也可能对大数据产生相当大的影响。根据IDC 2016年9月的报告，“31.4％的受访公司推出了物联网解决方案，另有43％希望在未来12个月内部署物联网解决方案。”

随着这些新设备和应用程序上线，许多公司需要新的技术和系统，才能够处理和感知来自物联网的大量数据。

边缘计算是一种可以帮助公司处理物联网大数据的新技术。在边缘计算中，大数据分析非常接近物联网设备和传感器，而不是数据中心或云。对于企业来说，这种方式的优点显而易见。因为在网络上流动的数据较少，可以提高网络性能并节省云计算成本。它还允许公司删除过期的和无价值的物联网数据，从而降低存储和基础架构成本。边缘计算还可以加快分析过程，使决策者能够更快地洞察情况并采取行动。

对于IT工作者来说，大数据的发展意味着大数据技能人才的高需求。IDC称，“到2018年，美国将有181,000个深度分析岗位，是数据管理和数据解读相关技能岗位数量的五倍。”

由于人才缺口过大，罗伯特·哈夫技术公司预测，到2017年数据科学家的平均薪资将增长6.5%，年薪在116,000美元到163,500美元之间（当然这是美国的标准，中国国内目前尚未统计）。同样，明年大数据工程师的薪资也将增长5.8%，在135,000美元到196,000美元之间。

由于聘请高级专家的成本过高，许多公司开始转向数据分析工具。IDC先前预测，“视觉数据发现工具的增长速度将比其他商业智能（BI）市场快2.5倍，到2018年，所有企业都将投资终端用户自助服务。

一些大数据供应商已经推出了具有“自助服务”能力的大数据分析工具，专家预计这种趋势将持续到2017年及以后。 数据分析过程中，信息技术的参与将越来越少，大数据分析将越来越多地融入到所有部门工作人员的工作方式之中。

Top 10 Trends in Big Data

"Big data" is no longer just a buzzword. Researchers at Forrester have "found that, in 2016, almost 40 percent of firms are implementing and expanding big data technology adoption. Another 30 percent are planning to adopt big data in the next 12 months."

Similarly, the Big Data Executive Survey 2016 from NewVantage Partners found that 62.5 percent of firms now have at least one big data project in production, and only 5.4 percent of organizations have no big data initiatives planned or underway.

Researchers say the adoption of big data technologies is unlikely to slow anytime soon. IDC predicts that the big data and business analytics market will increase from $130.1 billion this year to more than $203 billion in 2020. "The availability of data, a new generation of technology, and a cultural shift toward data-driven decision making continue to drive demand for big data and analytics technology and services," said Dan Vesset, group vice president, analytics and information management. "This market is forecast to grow 11.3 percent in 2016 after revenues reached $122 billion worldwide in 2015 and is expected to continue at a compound annual growth rate (CAGR) of 11.7 percent through 2020."

While it's clear that the big data market will grow, how organizations will be using their big data is a little less clear. New big data technologies are entering the market, while use of some older technologies continues to grow. This slideshow covers ten top trends that will likely shape the big data market in 2017 and beyond.

Big Data Trends

Experts expect machine learning, predictive analytics, IoT and edge computing to have a big impact on big data projects in 2017 and beyond.

1. Open Source

Open source applications like Apache Hadoop, Spark and others have come to dominate the big data space, and that trend looks likely to continue. One survey found that nearly 60 percent of enterprises expect to have Hadoop clusters running in production by the end of this year. And according to Forrester, Hadoop usage is increasing 32.9 percent per year.

Experts say that in 2017, many enterprises will expand their use of Hadoop and NoSQL technologies, as well as looking for ways to speed up their big data processing. Many will be seeking technologies that allow them to access and respond to data in real time.

2. In-Memory Technology

One of the technologies that companies are investigating in an attempt to speed their big data processing is in-memory technology. In a traditional database, the data is stored in storage systems equipped with hard drives or solid state drives (SSDs). In-memory technology stores the data in RAM instead, which is many, many times faster. A report from Forrester Research forecasts that in-memory data fabric will grow 29.2 percent per year.

Several different vendors offer in-memory database technology, notably SAP, IBM, Pivotal.

Image Source: Micron Technology

3. Machine Learning

As big data analytics capabilities have progressed, some enterprises have begun investing in machine learning (ML). Machine learning is a branch of artificial intelligence that focuses on allowing computers to learn new things without being explicitly programmed. In other words, it analyzes existing big data stores to come to conclusions which change how the application behaves.

According to Gartner machine learning is one of the top 10 strategic technology trends for 2017. It noted that today's most advanced machine learning and artificial intelligence systems are moving "beyond traditional rule-based algorithms to create systems that understand, learn, predict, adapt and potentially operate autonomously."

Image Source: MapR

4. Predictive Analytics

Predictive analytics is closely related to machine learning; in fact, ML systems often provide the engines for predictive analytics software. In the early days of big data analytics, organizations were looking back at their data to see what happened and then later they started using their analytics tools to investigate why those things happened. Predictive analytics goes one step further, using the big data analysis to predict what will happen in the future.

The number of organizations using predictive analytics today is surprisingly low—only 29 percent according to a 2016 survey from PwC. However, numerous vendors have recently come out with predictive analytics tools, so that number could skyrocket in the coming years as businesses become more aware of this powerful tool.

Image Source: Gartner

5. Intelligent Apps

Another way that enterprises are using machine learning and AI technologies is to create intelligent apps. These applications often incorporate big data analytics, analyzing users' previous behaviors in order to provide personalization and better service. One example that has become very familiar is the recommendation engines that now power many ecommerce and entertainment apps.

In its list of Top 10 Strategic Technology Trends for 2017, Gartner listed intelligent apps second. "Over the next 10 years, virtually every app, application and service will incorporate some level of AI," said David Cearley, vice president and Gartner Fellow. "This will form a long-term trend that will continually evolve and expand the application of AI and machine learning for apps and services."

Image Source: Microsoft

6. Intelligent Security

Many enterprises are also incorporating big data analytics into their security strategy. Organizations' security log data provides a treasure trove of information about past cyberattack attempts that organizations can use to predict, prevent and mitigate future attempts. As a result, some organizations are integrating their security information and event management (SIEM) software with big data platforms like Hadoop. Others are turning to security vendors whose products incorporate big data analytics capabilities.

Image Source: IBM

7. IoT

The Internet of Things is also likely to have a sizable impact on big data. According to a September 2016 report from IDC, "31.4 percent of organizations surveyed have launched IoT solutions, with an additional 43 percent looking to deploy in the next 12 months."

With all those new devices and applications coming online, organizations are going to experience even faster data growth than they have experienced in the past. Many will need new technologies and systems in order to be able to handle and make sense of the flood of big data coming from their IoT deployments.

Image Source: Verizon State of the Market: Internet of Things 2016

8. Edge Computing

One new technology that could help companies deal with their IoT big data is edge computing. In edge computing, the big data analysis happens very close to the IoT devices and sensors instead of in a data center or the cloud. For enterprises, this offers some significant benefits. They have less data flowing over their networks, which can improve performance and save on cloud computing costs. It allows organizations to delete IoT data that is only valuable for a limited amount of time, reducing storage and infrastructure costs. Edge computing can also speed up the analysis process, allowing decision makers to take action on insights faster than before.

Image Source: Dell.com

9. High Salaries

For IT workers, the increase in big data analytics will likely mean high demand and high salaries for those with big data skills. According to IDC, "In the U.S. alone there will be 181,000 deep analytics roles in 2018 and five times that many positions requiring related skills in data management and interpretation."

As a result of that scarcity, Robert Half Technology predicts that average compensation for data scientists will increase 6.5 percent in 2017 and range from $116,000 to $163,500. Similarly, big data engineers should see pay increases of 5.8 percent with salaries ranging from $135,000 to $196,000 for next year.

Image Source: Robert Half Technology 2017 Salary Guide for Technology Professionals

10. Self-Service

As the cost of hiring big experts rises, many organizations are likely to be looking for tools that allow regular business professionals to meet their own big data analytics needs. IDC has previously predicted "Visual data discovery tools will be growing 2.5 times faster than rest of the business intelligence (BI) market. By 2018, investing in this enabler of end-user self service will become a requirement for all enterprises."

Several vendors have already launched big data analytics tools with "self-service" capabilities, and experts expect that trend to continue into 2017 and beyond. IT is likely to become less involved in the process as big data analytics becomes more integrated into the ways that people in all parts of the business do their jobs.

来源：灯塔大数据；微信：DTbigdat