![]() ![]() For example, if you send an Amazon Kinesis record to a Splunk index without formatting the record first, the indexed record displays the following metadata: Since the Amazon Kinesis records don't contain values for these fields, these fields become populated with default metadata. When you use the Send to Splunk HTTP Event Collector sink function to send data from a DSP pipeline to a Splunk index, the sink function maps these standard fields to analogous fields in the resulting indexed records. As a result, the Splunk platform indexes these records as empty events.Īdditionally, Amazon Kinesis records don't include the timestamp, host, source, and source_type fields, which are part of the standard DSP schemas. The Amazon Kinesis Data Stream source function outputs records that use the following schema:Īmazon Kinesis records don't contain the body field, since their payloads are stored in the value field instead. Problems with indexing unprocessed data from Amazon Kinesis Data Streams The data values in the records are human-readable.įor an example of how to build a custom pipeline that completes the necessary data processing, see the Example: Send data from Amazon Kinesis Data Streams to the Splunk platform using the section on this page.Any important pieces of metadata, such as the timestamp or source type associated with the record, are stored in the following top-level fields: timestamp, source_type, host, and source.The payload of the record is stored in a top-level field named body.In some cases, the value field in the record contains data values that are not human-readable, such as Gzip-compressed data.īefore sending Amazon Kinesis data from a DSP data pipeline to a Splunk index, make sure to format your records to meet the following criteria:.For example, the timestamp in the indexed record indicates the time when the event was ingested into the rather than the time when the event was actually generated. ![]() Some of the metadata fields contain values that pertain to your use of the DSP pipeline and the Splunk HTTP Event Collector (HEC) rather than the actual log or event from Amazon Kinesis Data Streams.The records are indexed as empty events that have associated metadata but no payload.If you send Amazon Kinesis records to an index without formatting the records first, you'll notice problems such as the following: When you use the Amazon Kinesis Data Stream source function to receive data, the output records use a schema that can't be indexed meaningfully in the Splunk platform. Formatting data from Amazon Kinesis Data Streams for indexing in the Splunk platform ![]()
0 Comments
Leave a Reply. |