Porphyrio is a reliable data partner for the poultry industry. By developing cloud-based IoT and big data solutions, Porphyrio enables each customer to optimize their business, online and in real-time. As such, their customers can take better strategic decisions.
A massive amount of both manual and automatic data is entering the Porphyrio system each day. The data describes the daily processes in the poultry industry. It contains information about the environment, such as temperature, humidity, and numerous other gas densities. But also the number of eggs, the number of newborn poultry en the number of deaths are gathered by the system with a fixed frequency.
However, the data may contain noise and other artifacts causing systems to crash or to output strange results. Detecting these errors and preventing them from happening in the future is essential in delivering a reliable and accurate platform to the support managers.
Together with Porphyrio, Sentigrate developed a data pipeline to reliably and accurately ingest all sorts of data into their system.
A combination of Nifi and Airflow allowed us to programmatically define the data flow and corresponding steps in Python. This way the data can be transformed, manipulated and cleaned in a testable and extensible way, allowing for a clear separation of concern for each step.
The results of each intermediate step were saved to a separate table in a Postgres database, which ensured easier debugging but also allowed us to easily resume the data flow when necessary.
Number and facts about the case
Python, Airflow, Nifi
Mei 2017 - August 2017
5-10 employees
Acquired by Evonik