Last week, we had a fun Delta Lake 0. 7. 0 + Apache Spark 3. 0 AMA where Burak Yavuz,…
Continue Readinglake
A Guide to the Databricks + AWS Cloud Data Lake Dev Day Workshop
The Databricks team has been working hard to recreate content and enhance the experience as we transition all our events…
Continue ReadingModern Industrial IoT Analytics on Azure – Part 2
Introduction In part 1 of the series on Modern Industrial Internet of Things (IoT) Analytics on Azure, we walked through…
Continue ReadingMonitor Your Databricks Workspace with Audit Logs
Cloud computing has fundamentally changed how companies operate – users are no longer subject to the restrictions of on-premises hardware…
Continue ReadingSchema Evolution in Merge Operations and Operational Metrics in Delta Lake
Try this notebook to reproduce the steps outlined below We recently announced the release of Delta Lake 0. 6. 0,…
Continue ReadingHow to Prevent Data Black Holes from Swallowing your Organization Whole
In this special guest feature, Tolga Tarhan, Chief Technology Officer at Onica, points out that as data accumulates in an…
Continue ReadingBuilding a Modern Clinical Health Data Lake with Delta Lake
The healthcare industry is one of the biggest producers of data. In fact, the average healthcare organization is sitting on…
Continue ReadingSecurity that Unblocks the True Potential of your Data Lake
Over the last few years, Databricks has gained a lot of experience deploying data analytics at scale in the enterprise.…
Continue ReadingQuery Delta Lake Tables from Presto and Athena, Improved Operations Concurrency, and Merge performance
We are excited to announce the release of Delta Lake 0. 5. 0, which introduces Presto/Athena support and improved concurrency.…
Continue ReadingSchema Evolution in Data Lakes
By Hussein Danish, Data Engineer @ SSENSE There are countless articles to be found online debating the pros and cons of…
Continue ReadingMake Your Data Lake CCPA Compliant with a Unified Approach to Data and Analytics
CCPA requires businesses to potentially delete all personal information about a consumer upon request. Many organizations today are using or…
Continue ReadingDatabricks Demonstrates AWS Platform Integrations at re:Invent 2019
Session: Building Reliable Data Lakes for Analytics with Delta LakeIn this session, Michael Armbrust, the creator of Delta Lake, walked…
Continue ReadingAzure Databricks Highlights Adoption of Delta Lake, MLflow, and Integration with Azure Machine Learning at Microsoft Ignite 2019
It was an action-packed week of making new connections and learning about new innovation across data science, data engineering, and…
Continue ReadingDo You Actually Need a Data Lake?
Data lakes have become the cornerstone of many big data initiatives, just as they offer easier and more flexible options…
Continue ReadingAutomate and Fast-track Data Lake and Cloud ETL with Databricks and StreamSets
The challenges discussed above can slow down an organization’s cloud analytics/data science plans significantly – especially if they are limited…
Continue ReadingOkera Delivers Industry’s First Real-Time Actionable Insights into Data Lakes
Okera, a leading active data management platform that enables companies to discover, audit, and protect data at scale, announced Okera…
Continue ReadingScalable near real-time S3 access logging analytics with Apache Spark and Delta Lake
The original blog is from Viacheslav Inozemtsev, Senior Data Engineer at Zalando, reproduced with permission. Introduction Many organizations use AWS…
Continue ReadingHope is Not a Strategy for Deriving Value from a Data Lake
The good news is that, today, it doesn’t have to be that way. The data lake has come a long…
Continue ReadingDiving Into Delta Lake: Unpacking The Transaction Log
Breaking Down Transactions Into Atomic Commits Whenever a user performs an operation to modify a table (such as an INSERT,…
Continue ReadingProductionizing Machine Learning with Delta Lake
Try out this notebook series in Databricks – part 1 (Delta Lake), part 2 (Delta Lake + ML) For many…
Continue ReadingWhat Happened to Hadoop? And Where Do We Go from Here?
Monte Zweben, CEO of Splice Machine, has an interesting take on what happened to Hadoop, specifically three main reasons behind…
Continue ReadingMigrating Transactional Data to a Delta Lake using AWS DMS
AWS DMS can migrate your data from the most widely used commercial and open-source databases to S3 for both migrations…
Continue ReadingGetting Data Ready for Data Science: On-Demand Webinar and Q&A Now Available
On June 25th, our team hosted a live webinar — Getting Data Ready for Data Science — with Prakash Chockalingam,…
Continue ReadingSimplifying Streaming Stock Analysis using Delta Lake and Apache Spark: On-Demand Webinar and FAQ Now Available!
Traditionally, real-time analysis of stock data was a complicated endeavor due to the complexities of maintaining a streaming system and…
Continue ReadingCreating a data lake for GTFS RealTime data using AWS services: collection, storage and processing
Creating a data lake for GTFS RealTime data using AWS services: collection, storage and processingArash KavianiBlockedUnblockFollowFollowingMay 30As part of a…
Continue Reading