I ran a six-step pipeline based on TPC-DS, including fact-dimension joins and aggregations, reading from and writing to S3, on the three engines. All were run on default settings with no tuning and no caching between runs.

DuckDB 2.0 beat both Spark and StarRocks by a wide margin, in part because of its excellent new asynchronous I/O.

Full write-up of the details here on my project site:

https://www.duckstring.com/blog/duckdb-pipeline-engine

We've managed with hourly Snowflake loads for years and honestly they've been good enough most of the time. Lately though, product keeps asking for live dashboards while our ML team is pushing for fresher training data.

I'm not against moving to real-time, but I also don't want to introduce streaming infrastructure just because it sounds modern.

For anyone who's made the switch, what was the moment where real-time vs batch processing stopped being a debate and became an obvious decision?

Pre-mortem exercise: assume a nightly load has failed badly six months from now, and write the headlines. These are hypothetical, but I think each one is common. Next to each is the cheapest check I’d add.

1. “Revenue is double on Tuesdays”
Likely cause: a rerun after a failure inserted the same rows again.
Check: make the load idempotent (merge/upsert on a key, or replace a whole partition), and compare row counts by key after each run.

2. “Yesterday’s numbers are missing”
Likely cause: the source delivered late, and the job ran on whatever was there.
Check: a freshness check on the source (latest timestamp or file arrival) before the job starts, with a clear fail or wait.

3. “The job succeeded, but every value in a column is NULL”
Likely cause: the upstream renamed a column, and a lenient parser filled it with NULL.
Check: validate the schema (names and types) at ingest, and alert if the null rate of a column jumps compared with the previous runs.

4. “The load succeeded with zero rows”
Likely cause: an empty file or an empty query result counted as success.
Check: a minimum row count, or a comparison with the recent average, as a gate between extract and load.

5. “Daily totals are off by a few percent near midnight”
Likely cause: the source stores UTC, the report assumes local time (or the reverse), so some rows fall on the wrong day.
Check: state the time zone of each timestamp column in the pipeline, and test the day boundary with rows just before and after midnight.

6. “A backfill overwrote corrected data”
Likely cause: the backfill reloaded history from the source, replacing manual fixes made downstream.
Check: keep a dry-run mode that reports what would change, and require a row-level diff or count of changed rows before the write.

A pattern across these: most checks are cheap (a count, a null rate, a max timestamp), and most failures were silent, not loud. A job that fails loudly is easier than one that succeeds with wrong data.

Where I’d stop: I wouldn’t add every check to every table. Pick the few tables whose wrong numbers would cause the most damage, and start there.

Which headline would you add, and what check would have caught it?

Hey Guys!,

I work at Hevo, where we build a platform for moving data from databases, SaaS apps, and other sources into data warehouses.

A lot of the problems we work around are things like pipelines failing, source schemas changing unexpectedly, data arriving late, troubleshooting sync issues, and keeping track of what's happening across different pipelines. We also have built-in transformations and monitoring to help teams manage these workflows.

I wanted to open up a discussion here because I'd like to hear what people are actually dealing with in their day-to-day work.

Are you spending too much time maintaining connectors? Running into problems with CDC? Dealing with unexpected costs, slow syncs, or support issues when something breaks?

Feel free to ask me anything about Hevo, how certain things work, or where it might fit into your setup. And if you've used Hevo before and had a frustrating experience, I'd be interested in hearing about that too.

I'll answer what I can here, and I'll be honest if something isn't a good fit for Hevo.

What are you struggling with most in your current data setup?

Hi everyone,

CSV files exported with a naive comma join cells approach are a pain.

This typicly introduces split cells and row breaks due to commas and line returns which should have been encapsulated in a cell value.

I'm looking for an algorithm that will auto-detect and fix these type of RFC 4180 non-compliance

This can be a challenging problem as what fixes a file can also be what breaks another and I need something a higly reliable .

I'm looking for a solution that has a proven track record of solving real-world problems. This is where I was hoping you guys would help! I'm sure many of your pre-import cleaning pipelines have what I'm looking for!

Here is my current intuition on how to do it.

• Count columns per row.
• If over 90% of rows have the same number of columns, then the 10% remaining most likely need fixing.
• Rows with more columns have an "extra separator " parsing issue,
• Rows with fewer columns are expected to come in pairs and tend to be less common due to line returns.

For Line Returns

• Check the rows come in pairs.
• Check the column sum of 2 rows of a pair is equal to the expected number of columns.
• Merge the 2 rows.

For Commas

• Build statistics on the number of characters in valid rows (count alpha chars, count nums, count special chars in column 1, 2, 3, etc.) .
• In the row outlier, merge each consecutive cell turn by turn and score how the result matches the expected statistics .
• Merge the 2 cells that gave the best score and align the rest.

Context

I want to add an on file open: auto fix optional feature to an open source CSV file editing app aimed at data experts: https://nanocell-csv.com/

I'm looking forward to hear from your experiences, edge cases, and any tools/code lirbraries worth investigating. Thanks!

Hi everyone,

I tried to address some of the issues that can arise when using Power BI’s standard incremental refresh approach, following the pattern documented by Microsoft.

With large partitioned fact tables, even a relatively small incremental batch can end up affecting more partitions than I would actually want to refresh.

So I experimented with separating change detection from partition allocation logic.

The idea is to keep a stable key → class_id mapping, while dynamically limiting new records to a configurable subset of classes based on the batch size.

I implemented two approaches entirely in T-SQL:

Rotating Window — new records are allocated within a window of classes that rotates over time.
GitHub: https://github.com/Salvatore97710/rotating-window-partition-refresh

Least-Loaded — new records are allocated to the classes that currently contain the fewest records.
GitHub: https://github.com/Salvatore97710/least-loaded-class-partition-refresh

I’m curious to hear how people working with ETL pipelines would approach the same problem.

What would you change or do differently?

Hi r/ETL,

I'm the developer of Flowfile, a visual ETL tool built on Polars. I posted about it here a few months ago, but now it truly does two way design!

One of the things I ran into working with visual low-code ETL tools is the costs of switching between code <-> canvas. The latest releases of Flowfile are exactly addressing that!

Flowfile now ships with a Python notebook that let's you edit your visual ETL in real time and get's updated when you change something on the canvas. Basically, write Polars like code and push it and the transformations are shown on the canvas. Other way around, when you add transformations to the canvas, it will show as a new code block in your notebook!

You can install it via pip (`pip install flowfile`) or the full application from the releases.

I'm figuring out if it covers many use-cases and where things can be improved. For example, how well does the canvas model fit your way of writing Python transformations in Polars. So let me know if you like it and even better, if you want to form the future of this way of working!

GitHub: https://github.com/edwardvaneechoud/Flowfile

Most incremental loads I see described work like this: store the max updated_at, then pull WHERE updated_at > :watermark on the next run. It works for a long time, which is why the failure cases are easy to miss. Curious how others handle them. Here are the three I keep running into in the literature and docs, and the pattern I’d reach for.

Assumption 1: “Rows become visible in updated_at order.”
Not necessarily. In Postgres, now() is the transaction start time. A transaction that starts at 10:00:00 and commits at 10:00:07 stamps its rows 10:00:00. If your job ran at 10:00:03 and moved the watermark past that, those rows never show up in > watermark. Long transactions and batch updates make this more likely.

Assumption 2: “Timestamps from the source are comparable.”
If the app sets updated_at instead of the database, clocks on different servers can disagree by a bit. Precision matters too: if the source stores microseconds and your pipeline truncates to milliseconds, > can skip rows that share the truncated value, and >= can reprocess them.

Assumption 3: “The watermark sees everything that changed.”
Hard deletes leave no row behind, so updated_at can’t report them. Also, anything that updates a row without touching updated_at (a manual fix, a migration, a trigger that was never added) is invisible.

A pattern that covers most of this:

• Re-read a safety window instead of starting exactly at the watermark, e.g. WHERE updated_at > :watermark - interval '15 minutes'.
• Make the load idempotent (upsert/merge on the primary key), so reading the same row twice is harmless.
• Pick the window from your longest realistic transaction, not from a guess.
• For deletes, use soft-delete flags or CDC, or run a periodic reconciliation (count/checksum by day) against the source.
• If you can, use a log-based source (CDC) instead of timestamps. It removes the ordering problem at the cost of more infrastructure.

I’m not saying the simple watermark is wrong. For a small table with short transactions it’s often fine. But a periodic row-count comparison against the source is cheap, and it catches the quiet losses.

A pipeline finishes on schedule.

No errors. No failed jobs. Everything is green.

But what if a source system changed the meaning of a field?

What if duplicate events are being counted twice?

What if records are arriving late and yesterday's dashboard is missing 15% of the transactions?

Technically, the pipeline succeeded.

From the business's perspective, it failed.

I think this is why data engineering needs more than monitoring whether jobs complete.

You also need checks around freshness, completeness, uniqueness, schema changes, and important business rules.

And when something fails, the team should know which downstream reports or systems are affected.

For data engineers, which data quality checks have caught the most serious issues in your pipelines?

And which problems were hardest to detect because everything appeared to be running normally?

Hey all, disclosure first: I work at Altimate, which builds dbt Power User. We recorded one short video per workflow, 4 to 8 minutes each.

1. Profile Query: review a model's SQL for bugs and cost in one click. https://youtu.be/uMptd4t_mGI
2. Column-level lineage, with AI chat about the model. https://youtu.be/A-z_R0QJig0
3. Docs, data tests and unit tests in one editor. https://youtu.be/NrEYqEUcpHY
4. Explain, optimize and profile each CTE from inside the model. https://youtu.be/MDN2PSgVAmA
5. Hover over a ref() or macro to see what it does. https://youtu.be/kbRZ-hrENyc

Which of these would you like us to go deeper on?

[AI Disclosure] An AI was used to write the code for this program and to correct my english for this post

Hi everyone,

Tired of paying for GIS ETL tools, I decided to use my preferred LLM provider to code my own, and I’ve open-sourced it.

I’m happy to announce Carriole (French slang for “trailer”), a spatial-first ETL with a graphical interface, and to be honnest, I hope that it will be an alternative to FME for most of our use cases.

It’s currently in bêta, and I’m still looking for bugs and feedback from the community. The project is also open to contributions.

By advance, thanks for any of your feedback!

I recently started a new role as a Data Governance Manager at a mid size managed care organization. One of my first asks has been to create test templates for the new set of infrastructure tables. Essentially it’s to ensure all the tables that are now in production don’t have a glaring issue or whenever we make a change we go through the testing again.

My background is in data analytics and data management. I have very limited experience as a data engineer so aside from data quality tests I really don’t know what kind of tests should be run. I have used Claude to create a template of standard test cases but they seem either too convoluted or too simple like checking for nulls in non-nullable fields. Any resources or advice on how to ramp up or get a better understanding of this?

imagine spending hours tweaking Spark configs only to find out storage was the bottleneck lol.

AWS recently reported 31% faster Spark runtimes with S3 Express One Zone on EMR. been looking into Databricks tuning and tools like DualBird for speeding up ETL pipelines too.

anyone getting real gains or still fighting the same old shuffle headaches?

Last week I shared a benchmark comparing Scriptella, handwritten JDBC, csvsql, and PostgreSQL COPY for importing one million CSV rows.

A reader pointed out an important detail: PostgreSQL's JDBC driver defaults to reWriteBatchedInserts=false, while MariaDB Connector/J enables its bulk insert optimization by default.

So I reran the relevant cases, explicitly enabling and disabling PostgreSQL batch rewriting.

Results (median of 3 runs, batch size 1000):

Enabling rewriting reduced Scriptella's import time by 41.2% (1.70× faster) and handwritten JDBC's by 35.7% (1.55× faster).

The PostgreSQL-MariaDB gap for Scriptella shrank from 7.3 seconds to just 0.7 seconds.

No Scriptella implementation changes were involved. All 18 imports passed verification using the same CSV fixture, batch size, and transaction boundaries.

My main takeaway: Using JDBC driver defaults doesn't necessarily produce a meaningful cross-database comparison. Driver configuration can make a substantial difference.

The full benchmark still shows PostgreSQL COPY as significantly faster than JDBC batching, although COPY wasn't rerun in this focused local experiment. Comparing COPY against MariaDB's native LOAD DATA INFILE is another follow-up I'd like to explore.

I've also updated the benchmark's PostgreSQL configuration for future runs.

Full write-up, measurements, and reproduction details:

https://pvrlabs.xyz/java-performance/experiments/postgresql-batch-rewrite.html

I'd profile the Spark job before deciding whether to rewrite or scale. Shuffle overhead, data skew, spills, and CPU/memory utilization usually reveal where the bottleneck is.

I'm developing Hydra ETL, an open-source, YAML-driven ETL framework built around a related idea: separating transformation logic from the execution engine.

With pluggable engines like Spark, DuckDB, and Pandas, Hydra makes it possible to explore different execution strategies while keeping the same declarative pipeline definition.

It won't fix a poorly designed join, but it can reduce the cost of experimenting with alternatives.

Curious whether others have tried switching execution engines before committing to a major Spark rewrite.

https://hydraetl.com

https://www.youtube.com/watch?v=S9ACiT1dCJU&list=PLOVsdMXCxpho

https://github.com/bejaouibechir/Hydra

I’m curious if anyone here has contributed to Apache Spark.
For example, working on core modules or building/maintaining libraries, fixing issues, or submitting PRs.
I’d love to hear about your experience, what you worked on, and any advice for someone who wants to start contributing.

foorilla.com v1.11.3 | foorilla API v1.8.0 October 21, 2026

PRO+ provides full API access and filtering now at $96/month, while existing subscribers retain their access and pricing. PRO subscribers can now test the API using shared, frozen samples of up to 200 records per resource. The same API keys automatically follow subscription upgrades and downgrades. Samples support pagination and sampled detail IDs; full data access, filtering and anything else requires PRO+.

Related resources now use compact references instead of deeply nested objects. Nested topics no longer expand their entire tag collections, geographic references retain essential country/state context, and nested objects omit database timestamps, statistics, and embeddings. Meaningful data - including paper authors, categories, versions, and submission dates - remains included. Paper sample payloads are approximately 67% smaller, with OpenAPI documentation and frozen samples updated accordingly.

Breaking change: Nested response shapes have changed at the existing URLs; retrieve related resources from their own endpoints for full details. Resource IDs and existing PRO+ filtering and pagination behavior remain unchanged. Rate limits remain 6 requests/second and 360 requests/minute.