stream_microbatch_history
This table contains a history of every microbatch executed within this scheduler configuration.
Column | Data Type | Description |
---|---|---|
source_name | VARCHAR | The name of the source. |
source_cluster | VARCHAR | The name of the source cluster. The clusters are defined in stream_clusters. |
source_partition | INTEGER | The number of the data streaming partition. |
start_offset | INTEGER | The starting offset of the microbatch. |
end_offset | INTEGER | The ending offset of the microbatch. |
end_reason | VARCHAR |
An explanation for why the batch ended. The following are valid end reasons:
|
end_reason_message | VARCHAR | If the end reason is a network or source issue, this column contains a brief description of the issue. |
partition_bytes | INTEGER | The number of bytes transferred from a source partition to a Vertica target table. |
partition_messages | INTEGER | The number of messages transferred from a source partition to a Vertica target table. |
microbatch_id | INTEGER | The Vertica transaction id for the batch session. |
microbatch | VARCHAR | The name of the microbatch. |
target_schema | VARCHAR | The name of the target schema. |
target_table | VARCHAR | The name of the target table. |
timeslice | INTERVAL | The amount of time spent in the KafkaSource operator. |
batch_start | TIMESTAMP | The time the batch executed. |
batch_end | TIMESTAMP | The time the batch completed. |
last_batch_duration | INTERVAL | The length of time required to run the complete COPY statement. |
consecutive_error_count | INTEGER | (Currently not used.) The number of times a microbatch has encountered an error on an attempt to load. This value increases over multiple attempts. |
transaction_id | INTEGER | The identifier for the transaction within the session. |
frame_start | TIMESTAMP | The time the frame started. A frame can contain multiple microbatches. |
frame_end | TIMESTAMP | The time the frame completed. |
Examples
This example shows typical rows from the stream_microbatch_history table.
=> SELECT * FROM stream_config.stream_microbatch_history; -[ RECORD 1 ]--+--------------------------- source_name | streamsource1 source_cluster | kafka-1 source_partition | 0 start_offset | 196 end_offset | 196 end_reason | END_OF_STREAM partition_bytes | 0 partition_messages | 0 microbatch_id | 1 microbatch | mb_0 target_schema | public target_table | kafka_flex_0 timeslice | 00:00:09.892 batch_start | 2016-07-28 11:31:25.854221 batch_end | 2016-07-28 11:31:26.357942 last_batch_duration | 00:00:00.379826 consecutive_error_count | transaction_id | 45035996275130064 frame_start | 2016-07-28 11:31:25.751 frame_end | end_reason_message | -[ RECORD 2 ]--+--------------------------- source_name | streamsource1 source_cluster | kafka-1 source_partition | 1 start_offset | 197 end_offset | 197 end_reason | NETWORK_ISSUE partition_bytes | 0 partition_messages | 0 microbatch_id | 1 microbatch | mb_0 target_schema | public target_table | kafka_flex_0 timeslice | 00:00:09.897 batch_start | 2016-07-28 11:31:45.84898 batch_end | 2016-07-28 11:31:46.253367 last_batch_duration | 000:00:00.377796 consecutive_error_count | transaction_id | 45035996275130109 frame_start | 2016-07-28 11:31:45.751 frame_end | end_reason_message | Local: All brokers are down