Engineering Resource-Efficient Data Management for Smart Cities with Apache Kafka

In terms of the calibre and variety of services offered to end users, smart city management isundergoing a dramatic transformation. The parties involved in delivering pervasive applications cannow solve key issues in the big data value chain, including data gathering, analysis, and processing,storage, curation, and real-world data visualisation. This trend is being driven by Industry 4.0, whichcalls for the servitisation of data and products across all industries, including the field of smartcities, where people, sensors, and technology work closely together. In order to implement reactiveservices such as situational awareness, video surveillance, and geo-localisation while constantlypreserving the safety and privacy of affected persons, the data generated by omnipresent devicesneeds to be processed fast. This paper proposes a modular architecture to (i) leverage cutting-edgetechnologies for data acquisition, management, and distribution (such as Apache Kafka and ApacheNiFi); (ii) develop a multi-layer engineering solution for revealing valuable and hidden societalknowledge in the context of smart cities processing multi-modal, real-time, and heterogeneousdata flows; and (iii) address the key challenges in tasks involving complex data flows and offergeneral guidelines to solve them. In order to create an effective system for the monitoring andservitisation of smart city assets with a scalable platform that proves its usefulness in numeroussmart city use cases with various needs, we deduced some guidelines from an experimental settingperformed in collaboration with leading industrial technical departments. Ultimately, when deployedin production, the proposed data platform will contribute toward the goal of revealing valuable andhidden societal knowledge in the context of smart cities.