Node-RED automation for civil engineering insights
Project summary
Node-RED automation has transformed data processing from weeks into days by automating data ingestion, validation, cleaning, alignment, enrichment, and storage via a Node-RED pipeline along with Microsoft Fabric. For the business itself, that meant a greater data reliability, faster delivery, higher confidence, and accuracy.
They freed over 60% of time that had been allocated to routines to focus on priorities.
Services:
Project overview
Our client, a civil engineering company, was overwhelmed by large data volumes from numerous field sensors. The teams were burdened by inefficient manual processing, which shook analytical capabilities and confidence and delayed their projects.
We implemented a platform by leveraging Node-RED with Microsoft Fabric for automated data processing, which facilitated daily workflows by handling manual tasks and unlocked analytical power.
Main goals
- Minimize inefficient manual tasks to free up resources for high-value business processes
- Provide centralized data storage for instant data access
- Enable automated data processing – cleaning, resampling, filling, alignment, and more
- Enable real-time data integration and predictive analytics potential
The problem
A civil engineering team was struggling with gigabytes of time-series data extracted from their in-field sensors. All exported into bulky CSV and Excel files.
The skilled engineering team was spending an estimated 60% of their time on low-value, manual processes – cleaning, resampling, filling gaps, aligning timestamps.
The workflows were slow and inconsistent, which created extra bottlenecks and delayed important projects. What’s more, this chaos also undermined their capabilities and confidence in implemented analytical models.
In brief, the workflow was data-rich but insight-poor, which stopped the team from leveraging their resources and thought-out, strategic decision-making.
The solution
After joining this partnership, Abto Software successfully delivered a platform for automated data processing. Our engineers have replaced the error-prone, manual processes and introduced a drag-and-drop web console.
The selected low-code approach has allowed rapid deployment and enabled the team to modify the processes without possessing extensive expertise in programming.
A quick step-by-step breakdown
The platform is designed to support quick modifications of workflows without applying specialized expertise. The engineers just drop their raw data files into the intuitive application, then add custom nodes to handle data processing, and adjust the parameters in embedded Python scripts – the results are seen within seconds.
A pre-built visual pipeline – Node-RED powered – will execute the workflow, which includes:
- Data validation & cleaning, by detecting and correcting format issues and outliers
- Data alignment & enrichment, by resampling and syncing datasets across multiple sources
- Data visualization & storage
– by enabling instant analysis through generated flow-duration curves and hydrographs
– while storing structured datasets in the time-series database
Value delivered to business
By automating data processing, the team could free up over 60% of the time typically allocated to routines.
That means:
- More focus on priorities
- Data reliability
- Faster delivery
- Analytical accuracy
Initially aimed to resolve data bottlenecks, the platform has evolved to become a strategic business advantage. The solution is now being scaled to ingest IoT feeds in real-time and harness API integration.
This paves the way for predictive analytics services and opens up new revenue opportunities.
Node-RED automation: before vs after implementation
| Before automation | After automation | |
| Data ingestion | Manual collection and import of cumbersome CSV/Excel files | Automatic ingestion of raw data files |
| Data validation & cleaning | Manual review, often error-prone | Automatic detection and correction of errors, format issues, and outliers |
| Data alignment | Manual examination | Automatic resampling and synchronization across datasets |
| Data enrichment | Either limited or skipped | Automatic interpolation |
| Data visualization | Manual creation, often delayed | Automatic generation |
| Data storage | Dispersed and local files | A centralized Kusto database for quick data retrieval |
| Manual workload | ~60% of engineers’ time spent handling data processing | >60% of engineers’ time being available for high-priority business processes |
| Future scaling | Couldn’t handle large-scale integration | Can handle
|
Node-RED automation: use cases across domains
Scenario 1: real-time logistics & cold chain fleet monitoring
Food distributors are lacking real-time visibility, which causes serious problems – spoilage, disputes, and more. Node-RED automation might power real-time dashboards that include GPS tracking, temperature monitoring, and instant crisis alerts to minimize food wastage and associated business damage.
Scenario 2: smart building & energy management
Facility managers often face extreme expenses when dealing with disconnected HVAC and security systems. Node-RED automation might unify these systems to provide energy saving and predictive maintenance insights.
Scenario 3: laboratory equipment GXP monitoring & data integrity
Laboratory techs are risking being non-compliant when handling isolated instruments and making manual logs. Node-RED provides the opportunity to integrate all equipment and create one system with alerts and trails, thus ensuring real-time visibility.
Scenario 4: pharmaceutical manufacturing OEE monitoring & batch reporting
Production lines with mixed-vendor lab equipment are facing hidden inefficiencies and manual batch reporting. Node-RED aggregates data across all protocols and generates OEE dashboards with automated batch reports, thereby enabling better productivity and, naturally, regulatory compliance.
Technology stack: Node-RED for low-code projects
Node-RED is an open-source, flow-based tool that’s designed for event-driven data processing and automation. Its intuitive drag-and-drop interface is well-suited for prototyping and deploying enterprise-grade workflows without needing extensive coding.
For this specific project, Node-RED provided the balance of speed and function.
CLUTCH