Discover some of the most popular tools and technologies employers want for data engineering jobs, according to Dattell.
Demand for technology professionals has risen after a brief downturn following the onset of the COVID-19 pandemic. And a tech job with positions to fill across the industry is that of data engineer. Responsible for creating the infrastructure needed to consume, process and store massive amounts of information, data engineers are seeing an increase not only in job openings, but also in the tools that help them do their jobs.
Current data engineering job market
A report released on June 29, 2022 by data engineering provider Dattell looks at the current market for this career† To pool its findings, Dattell itself collected and analyzed a vast amount of data, tailoring specific technologies to the number of vacancies.
Dattell looked at the job market for data engineers and analyzed 340,000 different job openings. Of these, 35% were for positions in data orchestration, 30% for data storage, 29% for data visualization, and 6% for data processing.
In addition, Dattell analyzed the languages used by data engineers. Python was the most popular, preferred by employers 38% of the time, followed by Java at 33% and SQL at 22%. Together they account for 550,000 vacancies.
TO SEE: Hiring Kit: Database Engineer (Tech Republic Premium)
In addition, compensation for data engineering jobs ranged from $60,000 as a starting salary to as much as $180,000 for more advanced positions. The highest benefits were offered to those who work with Kubernetes, Elasticsearch, PostgreSQL, and Terraform, with salaries of $140,000 and above. And while there are quite a few openings for those who work with Tableau and Power BI, most offer less than $100,000.
Data Engineering Technology Trends
In his research, Dattell compiled a list of the 20 most popular data engineering technologies, citing tools such as Microsoft Power BI, Terraform, Chef, Spark, Elasticsearch, Hadoop, and Kafka. The products mentioned provide support in data storage, data orchestration, data processing and data visualization, indicating that no single segment of data engineering dominates the rest.
Based on Dattell’s research, the five most popular data engineering tools are MongoDB, Tableau, Kubernetes, PostgreSQL, and Ansible, which include data storage, data orchestration, and data visualization technologies.
Among the instruments themselves, tableau and Kubernetes took first places for the largest number of vacancies. Other tools that generated many vacancies were Ansible, Hadoop, Terraform, Splunk, Power BI, MongoDB and PostgreSQL.
And among the data orchestration tools studied, Kubernetes is by far the leader, followed by Ansible† While both products fall into the same space, each is used in a different way. Kubernetes allows professionals to manage and maintain container health, while Ansible allows them to make configuration changes and manage updates.
Rising popularity of free and open-source tools
With so many free products available, employers don’t see a great need to pay for data processing tools, Dattell said. As a result, companies are looking for employees and consultants with expertise in free and open-source technologies, such as leading tools such as Apache Spark and Apache Kafka.
For example, with data storage technologies, paid tools are preferred by 59% of employers, while a hefty 39% still prefer free tools. One of the most popular data storage technologies analyzed by Dattell, MongoDB was the most popular, followed by PostgreSQL, both of which are free. Some products, such as Elasticsearch, come in both paid and free versions.
