Every context's sidebar, |Hiring|, |Media|, |Research|, |Code|, |Meet| and |Work| as well as your Dashboard, now opens with a compact Pulse: each metric on one line with its count for the last 30 days, the change against the 30 days before, and a small daily trend underneath. When the earlier period is too small for a fair comparison, you'll see "low base" instead of a percentage, so no more swings like +11,200%. Below the Pulse, breakdowns show what matches your current filters (experience level, job type, research category, topic, language, event type, price and more), with up to the top 10 values each. Click any row to add it to your Context, or follow the Insight» link to explore the same area in Insight.

|Research| now has its own sidebar insights: newly submitted and revised papers, papers with linked code, and the categories and topics behind your current filters. |Work| gets them for the first time too, with new places and their mix by type, price and country. |Hiring| adds jobs with a stated salary, companies hiring, and the median stated salary for whatever you've filtered, together with its sample size. If too few matching jobs state a salary, we say so rather than hiding the figure.

The Stats tab on every record (jobs, companies, media items, sources, papers, repositories, updates, events, organizers and places) has been redesigned as well. It shows views and visits for the last 30 days with their change, a 13-week chart, and how the record ranks against comparable ones, for example "top 1% of jobs from the same week" or among papers in the same primary category.

Finally, Spaces in |Work| are now called Places: coworking spaces, cafés, libraries and other venues to get things done. Everything carries over automatically, including saved places and notes, saved Contexts and your history.

The API follows the rename in v1.9, which is a breaking change for Work integrations. /api/v1/work/space/, /api/v1/work/space/st/ and /api/v1/work/space/{pk} are now /api/v1/work/place/, /api/v1/work/place/st/ and /api/v1/work/place/{pk}, and the space_type field and filter is now place_type. API metadata uses place.* names, and geography statistics report place_count, place_view_count and place_click_count. The previous Space endpoints are no longer available, so please update any clients that use them. See the API documentation (/api/v1/docs) for the full endpoint and filter details.

I've been thinking about this while working with data pipelines, and I'm curious how other teams handle it.

Let's say data comes from different sources, lands in a staging area, goes through a bunch of transformations, and finally gets loaded into a warehouse where Bl reports use it.Where do you actually run your data quality checks?

For example, should we validate the data as soon as it arrives (nulls, duplicates, schema changes, data types, etc.)? Or should most of the checks happen after transformations, when we can validate the actual business logic?

The more I think about it, the more I feel that checking everything at every stage could make pipelines slower and harder to maintain. But if we only validate at the end, we might spend time debugging an issue that started much earlier in the pipeline.

A few things I'm particularly curious about:

At ingestion: Do you check schema, mandatory fields, duplicates, and unexpected data types before accepting the data?

After transformations: Do you validate business rules, joins, calculations, and whether records were unexpectedly dropped?

At the target: Do you compare source and target counts, totals, or individual records?Before BI reports: Do you run another set of checks, or do you trust the warehouse once the ETL pipeline passes?

When a check fails: Do you stop the entire pipeline, quarantine the bad records, or let the pipeline continue if the issue isn't critical?

I'm also wondering how you handle this when there are hundreds of tables and pipelines. Maintaining separate validation logic for every table sounds like it could become a lot of work.

Do you keep the checks inside the ETL code, use a separate data quality framework, or manage them as configuration-driven rules?

I'd love to hear how people structure this in real projects. Where do you draw the line between catching issues early and adding too many checks that become difficult to maintain?

Say your daily pipeline runs successfully, but a source system updates a record from three days ago. Your warehouse now has the old value, and your pipeline has already moved on.

How do you usually handle this?

Some approaches I've seen discussed are reprocessing a rolling window, using CDC or updated_at to pick up changed records, or running targeted backfills when a discrepancy is found.

I'm curious what actually works in production. Do you follow one approach across all your sources, or does it depend on the database or API? And what do you do when the source doesn't give you a reliable way to identify updated records?

I work at Hevo on the data integration side, and I'm interested in hearing how different teams handle this in practice.

I ran a six-step pipeline based on TPC-DS, including fact-dimension joins and aggregations, reading from and writing to S3, on the three engines. All were run on default settings with no tuning and no caching between runs.

DuckDB 2.0 beat both Spark and StarRocks by a wide margin, in part because of its excellent new asynchronous I/O.

Full write-up of the details here on my project site:

https://www.duckstring.com/blog/duckdb-pipeline-engine

Pre-mortem exercise: assume a nightly load has failed badly six months from now, and write the headlines. These are hypothetical, but I think each one is common. Next to each is the cheapest check I’d add.

1. “Revenue is double on Tuesdays”
Likely cause: a rerun after a failure inserted the same rows again.
Check: make the load idempotent (merge/upsert on a key, or replace a whole partition), and compare row counts by key after each run.

2. “Yesterday’s numbers are missing”
Likely cause: the source delivered late, and the job ran on whatever was there.
Check: a freshness check on the source (latest timestamp or file arrival) before the job starts, with a clear fail or wait.

3. “The job succeeded, but every value in a column is NULL”
Likely cause: the upstream renamed a column, and a lenient parser filled it with NULL.
Check: validate the schema (names and types) at ingest, and alert if the null rate of a column jumps compared with the previous runs.

4. “The load succeeded with zero rows”
Likely cause: an empty file or an empty query result counted as success.
Check: a minimum row count, or a comparison with the recent average, as a gate between extract and load.

5. “Daily totals are off by a few percent near midnight”
Likely cause: the source stores UTC, the report assumes local time (or the reverse), so some rows fall on the wrong day.
Check: state the time zone of each timestamp column in the pipeline, and test the day boundary with rows just before and after midnight.

6. “A backfill overwrote corrected data”
Likely cause: the backfill reloaded history from the source, replacing manual fixes made downstream.
Check: keep a dry-run mode that reports what would change, and require a row-level diff or count of changed rows before the write.

A pattern across these: most checks are cheap (a count, a null rate, a max timestamp), and most failures were silent, not loud. A job that fails loudly is easier than one that succeeds with wrong data.

Where I’d stop: I wouldn’t add every check to every table. Pick the few tables whose wrong numbers would cause the most damage, and start there.

Which headline would you add, and what check would have caught it?

We've managed with hourly Snowflake loads for years and honestly they've been good enough most of the time. Lately though, product keeps asking for live dashboards while our ML team is pushing for fresher training data.

I'm not against moving to real-time, but I also don't want to introduce streaming infrastructure just because it sounds modern.

For anyone who's made the switch, what was the moment where real-time vs batch processing stopped being a debate and became an obvious decision?

Hey Guys!,

I work at Hevo, where we build a platform for moving data from databases, SaaS apps, and other sources into data warehouses.

A lot of the problems we work around are things like pipelines failing, source schemas changing unexpectedly, data arriving late, troubleshooting sync issues, and keeping track of what's happening across different pipelines. We also have built-in transformations and monitoring to help teams manage these workflows.

I wanted to open up a discussion here because I'd like to hear what people are actually dealing with in their day-to-day work.

Are you spending too much time maintaining connectors? Running into problems with CDC? Dealing with unexpected costs, slow syncs, or support issues when something breaks?

Feel free to ask me anything about Hevo, how certain things work, or where it might fit into your setup. And if you've used Hevo before and had a frustrating experience, I'd be interested in hearing about that too.

I'll answer what I can here, and I'll be honest if something isn't a good fit for Hevo.

What are you struggling with most in your current data setup?

Hi everyone,

I tried to address some of the issues that can arise when using Power BI’s standard incremental refresh approach, following the pattern documented by Microsoft.

With large partitioned fact tables, even a relatively small incremental batch can end up affecting more partitions than I would actually want to refresh.

So I experimented with separating change detection from partition allocation logic.

The idea is to keep a stable key → class_id mapping, while dynamically limiting new records to a configurable subset of classes based on the batch size.

I implemented two approaches entirely in T-SQL:

Rotating Window — new records are allocated within a window of classes that rotates over time.
GitHub: https://github.com/Salvatore97710/rotating-window-partition-refresh

Least-Loaded — new records are allocated to the classes that currently contain the fewest records.
GitHub: https://github.com/Salvatore97710/least-loaded-class-partition-refresh

I’m curious to hear how people working with ETL pipelines would approach the same problem.

What would you change or do differently?

Hi everyone,

CSV files exported with a naive comma join cells approach are a pain.

This typicly introduces split cells and row breaks due to commas and line returns which should have been encapsulated in a cell value.

I'm looking for an algorithm that will auto-detect and fix these type of RFC 4180 non-compliance

This can be a challenging problem as what fixes a file can also be what breaks another and I need something a higly reliable .

I'm looking for a solution that has a proven track record of solving real-world problems. This is where I was hoping you guys would help! I'm sure many of your pre-import cleaning pipelines have what I'm looking for!

Here is my current intuition on how to do it.

• Count columns per row.
• If over 90% of rows have the same number of columns, then the 10% remaining most likely need fixing.
• Rows with more columns have an "extra separator " parsing issue,
• Rows with fewer columns are expected to come in pairs and tend to be less common due to line returns.

For Line Returns

• Check the rows come in pairs.
• Check the column sum of 2 rows of a pair is equal to the expected number of columns.
• Merge the 2 rows.

For Commas

• Build statistics on the number of characters in valid rows (count alpha chars, count nums, count special chars in column 1, 2, 3, etc.) .
• In the row outlier, merge each consecutive cell turn by turn and score how the result matches the expected statistics .
• Merge the 2 cells that gave the best score and align the rest.

Context

I want to add an on file open: auto fix optional feature to an open source CSV file editing app aimed at data experts: https://nanocell-csv.com/

I'm looking forward to hear from your experiences, edge cases, and any tools/code lirbraries worth investigating. Thanks!

Hi r/ETL,

I'm the developer of Flowfile, a visual ETL tool built on Polars. I posted about it here a few months ago, but now it truly does two way design!

One of the things I ran into working with visual low-code ETL tools is the costs of switching between code <-> canvas. The latest releases of Flowfile are exactly addressing that!

Flowfile now ships with a Python notebook that let's you edit your visual ETL in real time and get's updated when you change something on the canvas. Basically, write Polars like code and push it and the transformations are shown on the canvas. Other way around, when you add transformations to the canvas, it will show as a new code block in your notebook!

You can install it via pip (`pip install flowfile`) or the full application from the releases.

I'm figuring out if it covers many use-cases and where things can be improved. For example, how well does the canvas model fit your way of writing Python transformations in Polars. So let me know if you like it and even better, if you want to form the future of this way of working!

GitHub: https://github.com/edwardvaneechoud/Flowfile

Hey all, disclosure first: I work at Altimate, which builds dbt Power User. We recorded one short video per workflow, 4 to 8 minutes each.

1. Profile Query: review a model's SQL for bugs and cost in one click. https://youtu.be/uMptd4t_mGI
2. Column-level lineage, with AI chat about the model. https://youtu.be/A-z_R0QJig0
3. Docs, data tests and unit tests in one editor. https://youtu.be/NrEYqEUcpHY
4. Explain, optimize and profile each CTE from inside the model. https://youtu.be/MDN2PSgVAmA
5. Hover over a ref() or macro to see what it does. https://youtu.be/kbRZ-hrENyc

Which of these would you like us to go deeper on?

Most incremental loads I see described work like this: store the max updated_at, then pull WHERE updated_at > :watermark on the next run. It works for a long time, which is why the failure cases are easy to miss. Curious how others handle them. Here are the three I keep running into in the literature and docs, and the pattern I’d reach for.

Assumption 1: “Rows become visible in updated_at order.”
Not necessarily. In Postgres, now() is the transaction start time. A transaction that starts at 10:00:00 and commits at 10:00:07 stamps its rows 10:00:00. If your job ran at 10:00:03 and moved the watermark past that, those rows never show up in > watermark. Long transactions and batch updates make this more likely.

Assumption 2: “Timestamps from the source are comparable.”
If the app sets updated_at instead of the database, clocks on different servers can disagree by a bit. Precision matters too: if the source stores microseconds and your pipeline truncates to milliseconds, > can skip rows that share the truncated value, and >= can reprocess them.

Assumption 3: “The watermark sees everything that changed.”
Hard deletes leave no row behind, so updated_at can’t report them. Also, anything that updates a row without touching updated_at (a manual fix, a migration, a trigger that was never added) is invisible.

A pattern that covers most of this:

• Re-read a safety window instead of starting exactly at the watermark, e.g. WHERE updated_at > :watermark - interval '15 minutes'.
• Make the load idempotent (upsert/merge on the primary key), so reading the same row twice is harmless.
• Pick the window from your longest realistic transaction, not from a guess.
• For deletes, use soft-delete flags or CDC, or run a periodic reconciliation (count/checksum by day) against the source.
• If you can, use a log-based source (CDC) instead of timestamps. It removes the ordering problem at the cost of more infrastructure.

I’m not saying the simple watermark is wrong. For a small table with short transactions it’s often fine. But a periodic row-count comparison against the source is cheap, and it catches the quiet losses.

[AI Disclosure] An AI was used to write the code for this program and to correct my english for this post

Hi everyone,

Tired of paying for GIS ETL tools, I decided to use my preferred LLM provider to code my own, and I’ve open-sourced it.

I’m happy to announce Carriole (French slang for “trailer”), a spatial-first ETL with a graphical interface, and to be honnest, I hope that it will be an alternative to FME for most of our use cases.

It’s currently in bêta, and I’m still looking for bugs and feedback from the community. The project is also open to contributions.

By advance, thanks for any of your feedback!

A pipeline finishes on schedule.

No errors. No failed jobs. Everything is green.

But what if a source system changed the meaning of a field?

What if duplicate events are being counted twice?

What if records are arriving late and yesterday's dashboard is missing 15% of the transactions?

Technically, the pipeline succeeded.

From the business's perspective, it failed.

I think this is why data engineering needs more than monitoring whether jobs complete.

You also need checks around freshness, completeness, uniqueness, schema changes, and important business rules.

And when something fails, the team should know which downstream reports or systems are affected.

For data engineers, which data quality checks have caught the most serious issues in your pipelines?

And which problems were hardest to detect because everything appeared to be running normally?

I recently started a new role as a Data Governance Manager at a mid size managed care organization. One of my first asks has been to create test templates for the new set of infrastructure tables. Essentially it’s to ensure all the tables that are now in production don’t have a glaring issue or whenever we make a change we go through the testing again.

My background is in data analytics and data management. I have very limited experience as a data engineer so aside from data quality tests I really don’t know what kind of tests should be run. I have used Claude to create a template of standard test cases but they seem either too convoluted or too simple like checking for nulls in non-nullable fields. Any resources or advice on how to ramp up or get a better understanding of this?

Last week I shared a benchmark comparing Scriptella, handwritten JDBC, csvsql, and PostgreSQL COPY for importing one million CSV rows.

A reader pointed out an important detail: PostgreSQL's JDBC driver defaults to reWriteBatchedInserts=false, while MariaDB Connector/J enables its bulk insert optimization by default.

So I reran the relevant cases, explicitly enabling and disabling PostgreSQL batch rewriting.

Results (median of 3 runs, batch size 1000):

Enabling rewriting reduced Scriptella's import time by 41.2% (1.70× faster) and handwritten JDBC's by 35.7% (1.55× faster).

The PostgreSQL-MariaDB gap for Scriptella shrank from 7.3 seconds to just 0.7 seconds.

No Scriptella implementation changes were involved. All 18 imports passed verification using the same CSV fixture, batch size, and transaction boundaries.

My main takeaway: Using JDBC driver defaults doesn't necessarily produce a meaningful cross-database comparison. Driver configuration can make a substantial difference.

The full benchmark still shows PostgreSQL COPY as significantly faster than JDBC batching, although COPY wasn't rerun in this focused local experiment. Comparing COPY against MariaDB's native LOAD DATA INFILE is another follow-up I'd like to explore.

I've also updated the benchmark's PostgreSQL configuration for future runs.

Full write-up, measurements, and reproduction details:

https://pvrlabs.xyz/java-performance/experiments/postgresql-batch-rewrite.html

imagine spending hours tweaking Spark configs only to find out storage was the bottleneck lol.

AWS recently reported 31% faster Spark runtimes with S3 Express One Zone on EMR. been looking into Databricks tuning and tools like DualBird for speeding up ETL pipelines too.

anyone getting real gains or still fighting the same old shuffle headaches?