Open-source startups in Data engineering

Pipelines, orchestration and integration — the plumbing that moves data from where it lands to where it is used.

Ship status

10still shipping
0slowed
3gone silent

10 of 13 pushed in the last 30 days.

13 projects
firecrawlfirecrawl/firecrawlStill shipping

Knowledge infrastructure to make your agents smarter. Search, scrape and access 550+ data providers through one API.

Developers206.3KStill shipping
crawl4aiunclecode/crawl4aiStill shipping

Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.

Developers87.6KStill shipping
airbyteairbytehq/airbyteStill shipping

Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.

Data teamsalt to Fivetran22.4KStill shipping
pyspiderbinux/pyspiderGone silent

A Powerful Spider(Web Crawler) System in Python.

Developers16.7KGone silent
dagsterdagster-io/dagsterStill shipping

An orchestration platform for the development, production, and observation of data assets.

Data teams16.5KStill shipping
datahubdatahub-project/datahubStill shipping

The Context Platform for your Data and AI Stack

Data teams13KStill shipping
crawlabcrawlab-team/crawlabGone silent

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台,支持任何语言和框架

Developers12.3KGone silent
risingwaverisingwavelabs/risingwaveStill shipping

Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.

Developers9.4KStill shipping
mage-aimage-ai/mage-aiStill shipping

🧙 Build, run, and manage data pipelines for integrating and transforming data.

Data teams8.8KStill shipping
unstractZipstack/unstractStill shipping

LLM-Driven Extraction of Unstructured Data — Built for API Deployments & ETL Pipeline Workflows

Developers7.5KStill shipping
pachydermpachyderm/pachydermGone silent

Data-Centric Pipelines and Data Versioning

Data teams6.3KGone silent
lakeFStreeverse/lakeFSStill shipping

lakeFS - Data version control for your data lake | Git for data

Data teams5.6KStill shipping
jitsujitsucom/jitsuStill shipping

Jitsu is an open-source Segment alternative. Fully-scriptable data ingestion engine for modern data teams. Set-up a real-time data pipeline in minutes, not days

Data teamsalt to Segment5.1KStill shipping
How projects are grouped

Jev assigns one category from the repository name, description and homepage. Activity uses the last push to any branch.