Announcing the State Reader API: The New "Statestore" Data Source
Databricks Runtime 14.3 includes a new capability that allows users to access and analyze Structured Streaming 's internal state data: the State Reader...
Databricks Runtime 14.3引入了State Reader API,允许用户访问和分析Structured Streaming的内部状态数据。该API有助于开发、调试和故障排除有状态的Structured Streaming工作负载。它包括两个新的DataFrame格式选项:state-metadata和statestore。State Reader API简化了有状态流应用程序的开发和调试,提供了并行性和偏斜度的洞察,并帮助调查生产问题。该API已包含在Apache Spark 4.0.0中。