Open-source startups in Data engineering
Pipelines, orchestration and integration — the plumbing that moves data from where it lands to where it is used.
Ship status
10 of 13 pushed in the last 30 days.
Knowledge infrastructure to make your agents smarter. Search, scrape and access 550+ data providers through one API. | Developers | 206.3K | Still shipping | ||
Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it. | Developers | 87.6K | Still shipping | ||
Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud. | Data teamsalt to Fivetran | 22.4K | Still shipping | ||
A Powerful Spider(Web Crawler) System in Python. | Developers | 16.7K | Gone silent | ||
An orchestration platform for the development, production, and observation of data assets. | Data teams | 16.5K | Still shipping | ||
The Context Platform for your Data and AI Stack | Data teams | 13K | Still shipping | ||
Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台,支持任何语言和框架 | Developers | 12.3K | Gone silent | ||
Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale. | Developers | 9.4K | Still shipping | ||
🧙 Build, run, and manage data pipelines for integrating and transforming data. | Data teams | 8.8K | Still shipping | ||
LLM-Driven Extraction of Unstructured Data — Built for API Deployments & ETL Pipeline Workflows | Developers | 7.5K | Still shipping | ||
Data-Centric Pipelines and Data Versioning | Data teams | 6.3K | Gone silent | ||
lakeFS - Data version control for your data lake | Git for data | Data teams | 5.6K | Still shipping | ||
Jitsu is an open-source Segment alternative. Fully-scriptable data ingestion engine for modern data teams. Set-up a real-time data pipeline in minutes, not days | Data teamsalt to Segment | 5.1K | Still shipping |
How projects are grouped
Jev assigns one category from the repository name, description and homepage. Activity uses the last push to any branch.