In the railway and logistics sectors, organizations increasingly face the challenge of integrating large Excel files—often with hundreds of thousands of rows—into centralized systems with strict business logic. Many development projects struggle with this: the import process is too slow, doesn’t scale, or lacks proper error feedback. The typical scenario is a “blind upload,” where a faulty file halts the entire process without clearly identifying what went wrong.
We encountered the same issue in the ECMT (European Capacity Management Tool) project. The solution? We rebuilt the import mechanism from the ground up using our own technology stack—resulting in a scalable, transparent, and business-rule-driven process.
The Original Need: Integrating Data from Multiple Systems
Users wanted to import objects from external systems into ECMT. The earlier import solution could only handle files with around 10,000 rows—far less than what was required in real-world use cases.
How We Solved It
We built the new import system on our own Drupal-based platform, enhanced with Python-based backend components. This allowed us to:
- split files into smaller chunks and process them in parallel,
- validate business rules early in the pipeline,
- and reliably process up to 300,000-row files with structured feedback and high performance.
Core Components of the Solution
Business-Logic-Driven Validation
We check more than just technical correctness. Our import validates business-specific rules: user permissions, route definitions, valid dates, train type consistency, and more.
Optimized Internal Format
Instead of repeatedly accessing the original Excel file, we convert it to a custom internal format to boost processing speed.
Pre-Validation Feedback
Users receive immediate feedback if the uploaded file fails basic requirements (e.g., wrong template), saving system resources and user frustration.
Chunking and Parallel Processing
The file is split into smaller parts that are processed in parallel. If any chunk encounters an error, the process halts, and detailed error messages are displayed.
Unified Validation Logic
We reused the same validators used in the user interface to ensure consistent rule enforcement whether data is entered manually or via import.
What’s Next: Error Handling with AI Support
One of our planned development directions is to enhance the error-handling process with Large Language Model (LLM)-based AI assistance. The goal is to go beyond just identifying errors: we want to provide suggestions for corrections—or even handle them automatically in specific cases. This would make data processing faster and reduce support workload significantly.
Do you also struggle with Excel imports?
Let’s talk—whether you need a tailored solution or want to explore how AI can help future-proof your data workflows.
Share with your friends!