A massive amount of both manual and automatic data is entering the Porphyrio system each day. The data describes the daily processes in the poultry industry. It contains information about the environment, such as temperature, humidity, and numerous other gas densities. But also the number of eggs, the number of newborn poultry en the number of deaths are gathered by the system with a fixed frequency.

However, the data may contain noise and other artifacts causing systems to crash or to output strange results. Detecting these errors and preventing them from happening in the future is essential in delivering a reliable and accurate platform to the support managers.

Together with Porphyrio, Sentigrate developed a data pipeline to reliably and accurately ingest all sorts of data into their system.

A combination of Nifi and Airflow allowed us to programmatically define the data flow and corresponding steps in Python. This way the data can be transformed, manipulated and cleaned in a testable and extensible way, allowing for a clear separation of concern for each step.