Digital transformation has accelerated significantly due to the COVID-19 pandemic, but the extra demands on data scientists have revealed significant barriers to effective working and high levels of job dissatisfaction in some areas.

For example, around four in 10 are dissatisfied with their company’s use of analytics and model deployment, while more than 20 barriers to effective working emerged, according to a survey of data scientists commissioned by analytics leader SAS.

However, the work of data scientists has grown in importance with many organisations accelerating digital transformation projects by using technology to improve business operations.

More than 90% of respondents indicated the importance of their work was the same or greater compared to before the pandemic.

To delve deeper into the state of data science, the report assesses the impact of the pandemic, challenges faced, overall satisfaction with the analytics environment and more.

The research showed the pandemic upended standard business practices, shifting the assumptions and variables in models and predictive algorithms and causing a ripple effect of adaptations in processes, practices and operating parameters.

More than two-thirds of respondents were satisfied with the outcomes from analytical projects.

However, 42% of data scientists were dissatisfied with their company’s use of analytics and model deployment, suggesting a problem with how analytical insights are used by organisations to inform decision making.

This was backed up by 42% saying data science results were not used by business decision makers, making it one of the main barriers faced.

The survey also highlighted some specific skills gaps. Less than a third of the respondents reported having advanced or expert proficiency in program-heavy skills, such as cloud management and database administration.

This is an issue given that use of cloud services is up significantly, with 94% saying they experienced the same or greater use of cloud since COVID-19 struck.

“There have clearly been more demands placed on data scientists as the pandemic has accelerated digital transformation projects that many organisations were planning anyway,” said Dr. Iain Brown, head of Data Science at SAS UK and Ireland.

“A major source of frustration is finding a way for organisations to implement the insights from analytics projects and use them in their decision making, which means giving data scientists a seat at the boardroom table might be a way forward.”

“Linked to this, we found concerns around support for data science teams and a lack of talent, which has been an issue for some time with demand outstripping supply. Organisations must realise that investing in a team of data scientists with complementary skills could reap huge value for the business, so the cost of hiring needs to consider the return on that investment as we move to significantly more digital and AI-driven business processes,” continued Brown.

The research also identified gaps in consistent organisational emphasis on AI ethics, with 43% of respondents indicating that their organisation does not conduct specific reviews of its analytical processes with respect to bias and discrimination and only 26% of respondents reporting that unfair bias is used as a measure of model success in their organisation.

When it comes to the challenges identified to ensure fair and unbiased decision making, Dr. Sally Eaves, an industry expert, said, “Data scientists can lend their expertise to craft working guidelines for data access, usage security, and broader issues, such as sustainability and data ethics and bias.”

“Rather than sometimes hoping they are given appropriate, clean data and relying too much on the technology to drive fair outcomes, they can play an active role to put in place the right guidelines and checks at each stage of the analytical process to try and eliminate bias. Having a transparent and explainable flow from data to decision is obviously key to this,” said Eaves.

The research revealed positive outcomes from the global disruption of the pandemic. Nearly three-quarters (73%) said they are just as productive or more productive since the pandemic, while a similar proportion (77%) revealed they had the same or greater collaboration with colleagues. This suggests many of the challenges highlighted were in existence, possibly to a greater degree, before the pandemic.

Other challenges experienced were the amount of time spent on data preparation versus model creation. Respondents are spending more of their time (58%) than they would prefer gathering, exploring, managing and cleaning data.

“Overall, the data scientist has ample reason to feel empowered and optimistic about how the pandemic has shone a spotlight on the importance of their role within their organisation and how it might evolve over time,” said Brown. “This holds especially true if data scientists can leverage the whole spectrum of available tools to manage the analytics lifecycle, pursue data science training and skill development opportunities, and embrace data prep as the first step in modelling.”

Additional findings and discussion points, including strategies for data scientists to improve processes and outcomes, are available in the full report: Accelerating Digital Transformation.