An open source future for improved data management

Shell has joined Linux Foundation Energy (LF Energy) and is in the process of making a code contribution to mark the event. The code for the data ingestion element of Shell’s Sensor Intelligence Platform will be made open source under the name Real Time Data Ingestion Platform (RTDIP). This code has gone through years of development and provides time-series data management capability. By sharing it via LF Energy Shell hopes to encourage collaboration, accelerate innovation, and benefit organisations across the energy industry and beyond.

By Ian Betts on Oct 22, 2022


Until recently options for sourcing software tools and platforms were limited to buying licences for industry standard tools and attempting to customise them where needed, or to develop in-house products to meet precise needs but usually requiring large investment in development and maintenance. But there is another option. The open source model of shared development offers a viable “middle way” between purchasing standard products and creating bespoke systems.

Open source also offers a way to collaborate and support integration across value chains. Increased interoperability helps companies easily co-create solutions for frictionless flow of ideas and data. Furthermore, by enabling market access for smaller entities, open source can support rapid innovation.

Shell is already involved in several open source communities focused on various industry sectors. As customers and suppliers in the energy sector work ever more closely together, Shell sees a clear need to foster this collaboration to support the Energy Transition.

LF Energy supports cross company collaboration through open source, with a focus on electrification and power networks so they become equipped to cope with intermittent renewable and distributed energy generation.

Sharing the code – what is Shell bringing to the table?

The code Real Time Data Ingestion Platform (RTDIP) Shell will contribute is a foundational time-series data ingestion capability which today gathers data from more than three million sensors in Shell’s assets. Impressive as this is, Shell sees huge potential for use of this code in other areas that are rich in time-series data. Working with LF Energy is relevant to Shell’s emerging power business, and this capability has the potential to be deployed within the OSDUTM Data Platform, and Shell hopes it can play a role in further enhancement of the OSDUTM Data Platform’s utilization of data from renewable and other energy sources.

Anyone across or beyond the energy sector who needs to manage similar kinds of data may find value in these data ingestion capabilities. Shell sees opportunity for it to be used to monitor manufacturing or production processes, to track resource utilisation such as water usage, the operation of biorefineries or hydrogen plants and carbon capture and storage (CCS) projects. Time-series data is also extensively used to forecast solar and wind energy production where gathering insights from such data helps inform decision making for new or existing renewable energy projects.

Eyes on a major prize

Opportunities for collaboration through LF Energy could help Shell and others develop new and better ways to manage power. Shell anticipates efficiency gains from breaking down traditional data format barriers that exist between companies, sectors, and national jurisdictions. Some of the most obvious benefits Shell hopes for are in renewable energy production and distribution. Shell produces electricity from a vast range of wind and solar sources across many different countries. Adopting an open source approach with LF Energy will help better integrate these resources into distribution grids and build towards a more efficient and effective digital energy infrastructure.

Open source ecosystems also encourage new collaborations. By creating a common data fabric with the member companies of LF Energy, Shell hopes to deal more effectively with microgrids and help bring renewable power sources to the grid more easily. Bringing data together in common formats makes it easier for grid operating companies to work together and this contribution will create a shared data fabric that helps enable Shell to leverage software capabilities in an interoperable fashion while minimizing underlying data complexity.

The way ahead

This code contribution marks another step in Shell’s and the energy industries journey towards shared, open source development of tools that help enable better data management. Collaborating under the LF Energy banner helps show consistency of purpose and commitment to continued innovation. It underlines Shell’s determination to assist with developing an open source data fabric that will help transform and improve energy sector operations.

Read more about the Real Time Data Ingestion Platform on LF Energy 

*Shell International Exploration and Production Inc


The companies in which Shell plc directly and indirectly owns investments are separate legal entities. In this document “Shell” is sometimes used for convenience where references are made to Shell plc and its subsidiaries in general.

More in Digitalisation

Artificial Intelligence

From machine learning to computer vision, deep learning to virtual assistants and autonomous vehicles to robotics, Shell has been focused on a range of technologies that have supported advances in AI.

Digitalisation in action

Digital technologies are improving the experience of our customers and making Shell a more efficient business. They are increasing the effectiveness and efficiency of our processes by collecting quality data, turning this data into insights and insights into actions that improve the performance of our assets.

Digitalisation Research Publications

Shell is built on 125 years of technological innovation. Our grasp of computational technology helped us to lead the way in technological developments in exploration in the 1960s, 70s and 80s. This expertise in mathematics and computing is what gives us such a strong advantage today in developing and adopting digital technology.