Avin Jain’s Post

Three things to remember about Cortex: 1. It provides a familiar UI for your Cherwell data 2. It requires NO database changes 3. It requires minimal configuration In today’s video, we show the second one in probably the most ridiculous demo we’ve ever done. NO data changes. NO export/import, NO ETL, NO CSV, yet ALL of the data! Cortex (Cherwell Archive) https://lnkd.in/g8qycQ85 #cherwell #cherwellservicemanagement

🚀 ETL - Data Quality Check! 🚀 Below are some high level metrics which need to be consider 1️⃣ Historical Data Check 🕰️ 2️⃣ Incremental Load Verification 🆕 3️⃣ Transformation Accuracy 🔍 4️⃣ Referential Integrity Maintenance 🔗 5️⃣ Latency Monitoring ⏱️ 6️⃣ Error & Rejection Management ⚠️ 7️⃣ Backup & Archive Verification 🗃️ 8️⃣ Security & Compliance Adherence 🛡️ 9️⃣ Performance Optimization 🚀 🔟 Alerting & Monitoring Functionality 🚨 1️⃣1️⃣ Volume Management 📊 1️⃣2️⃣ Data Reconciliation ✅ 1️⃣3️⃣ Usability Assurance 🖥️ 1️⃣4️⃣ Documentation & Metadata Accuracy 📝 🔗 #DataEngineering #ETL #DataQuality #DataMigration #DataManagement

4 ways to make a Senior Data Engineer cry: 1. Disregard Data Quality: No validation, 0 cleansings, No Dupe check. 2. No Reusable Framework: Script every ETL process afresh, who needs efficiency? 3. Ignore Speed: Use single-threaded processes, distributed big data processing is overrated. 4. Bypass Monitoring: No failure alerts in the pipeline. Who needs more emails? (Bonus) - Tell the business all the data for the new dashboard will be ready in a day. #DataEngineeringBlues 😢 #DataEngineering Inspired by the kickass format from Luca Zanna

In our project, we've been facing a recurring ETL issue where the dynamic file headers in monthly CSV files cause our ETL pipelines to fail. During my POC on this problem, I encountered key data engineering concepts that will guide our architectural decisions based on our use cases: 1. Schema on write: This approach involves defining the schema before writing data to the database. It's used for structured data where the schema is known beforehand. 2. Schema on read: This approach involves defining the schema after writing data to the database. It's suitable for semi-structured data where the schema is not known in advance. #dataengineer #warehouse #innovasolutions #etlpipeline #bestpractices

So we've built this brand new ETL data pipeline to share low-voltage data with network operators and found this BIG GAP several months later. This pipeline is ingesting load-profile data (in batches) from the MDMS system in JSON format and then "processes and transforms" into the client-specific XML/CSV format. Good thing is: We capture the progress of the transformation job and have a UI to monitor the status and retrigger if the job fails. Big Gap: We don't show the load-profile data used in the job over the UI Result: The SIT Team spent too much time figuring out the correctness and wholeness of the output data versus the input data. Did we not know it? Yes, we did, and the MDP team too. But we ignored it, as nobody was concerned about the visibility of the input. Lesson Learned: When implementing transformations, make the inputs and outputs highly visible to keep everyone aligned. #smartmetering #evcharginginfrastructure #vpp #utilities #dataintegration #systemintegrators #energytransition

Are you an ETL Developer drowning in manual data validation? Automate time-consuming processes holding you back and regain your time. https://hubs.ly/Q02JdG0r0 #ETL #DataValidation #DataOpsSuite

Data Integration is a big world with lots of tools.

🔧🚀 Have you ever wondered how much impact optimized data pipelines can have? By refining ETL processes, I’ve seen processing times cut by up to 50%! Here’s what works: • Automated Testing: Catch issues before they escalate. • Incremental Loading: Save time and resources. • Data Quality Checks: Ensure your data is clean and reliable. What techniques do you use to enhance efficiency? #DataEngineering #Optimization

Data JPA Relationship (Entity Mapping) | Spring Boot 3.x Tutorial 3 https://lnkd.in/gvjZw5fp

𝐖𝐡𝐚𝐭 𝐡𝐚𝐬 𝐛𝐞𝐞𝐧 𝐭𝐡𝐞 𝐦𝐨𝐬𝐭 𝐜𝐡𝐚𝐥𝐥𝐞𝐧𝐠𝐢𝐧𝐠 𝐩𝐫𝐨𝐣𝐞𝐜𝐭 𝐨𝐫 𝐬𝐜𝐞𝐧𝐚𝐫𝐢𝐨 𝐲𝐨𝐮'𝐯𝐞 𝐟𝐚𝐜𝐞𝐝 𝐢𝐧 𝐲𝐨𝐮𝐫 𝐝𝐚𝐭𝐚 𝐜𝐚𝐫𝐞𝐞𝐫 𝐬𝐨 𝐟𝐚𝐫? 𝐋𝐞𝐭'𝐬 𝐥𝐞𝐚𝐫𝐧 𝐟𝐫𝐨𝐦 𝐞𝐚𝐜𝐡 𝐨𝐭𝐡𝐞𝐫! One of the most challenging projects I faced was building a real-time data pipeline to capture changes from a legacy on-premises database and replicate that data to a cloud-based data lake.The key challenge was dealing with the database's outdated technology stack and lack of native change data capture (CDC) capabilities. We couldn't just plug in a off-the-shelf CDC tool - we had to get creative.After evaluating a few options, we ended up using a combination of database triggers, custom ETL scripts, and a message queue to detect and propagate the changes. It was a complex, multi-step process, but it allowed us to keep the data flowing without disrupting the existing systems. What was once a slow, manual process became a living, breathing data ecosystem that fueled better decision-making.