DataPelago has emerged from stealth with a new platform that aims to optimize hardware usage for data-intensive workloads. The company’s “universal data processing engine” leverages all available computing resources, including CPUs, GPUs, and FPGAs, to process large volumes of structured, semi-structured, and unstructured data efficiently. The core of DataPelago’s technology consists of two intelligent layers: DataVM and DataOS.
DataVM uses a domain-specific instruction set for data operators and dynamically abstracts accelerated computing elements. DataOS maps these operations to the underlying processing resources to optimize performance. Co-founder and CEO Rajan Goyal emphasized the platform’s flexibility, stating that the system automatically decides where to run each workload based on the available hardware resources.
The goal is to deliver the promise of accelerated computing without requiring any changes to the application.
Datapelago’s universal data processing engine
DataPelago is working on integrating its platform with popular query engines and analytic frameworks such as Apache Trino and Apache Spark.
The company reports that its technology is currently being tested by clients looking to control hardware costs and accelerate AI model training. According to DataPelago, their engine can process data up to ten times faster than traditional computing platforms and at a significantly reduced cost. Benchmark results show that the platform is 3x to 4x faster than some of the fastest data lakehouse platforms available.
The company has raised $47 million in funding from investors such as Eclipse Ventures, Taiwania Capital, Qualcomm Ventures, Alter Venture Partners, Nautilus Ventures, and Silicon Valley Bank. With this financial backing, DataPelago aims to further develop and expand its platform to meet the growing demands for efficient hardware utilization in data processing and analytics. DataPelago targets various data-intensive workloads, including SQL queries, ETL jobs, streaming data workloads, and generative AI tasks.
The platform is scheduled for a full launch in the first quarter of 2025.
