Introduction

The log parser configuration involves setting up and customizing a log parser tool to extract, transform and analyze log data from various sources. Log parser configuration configures indexes to the UI side i.e.: input.conf,filter.conf and output.conf. To configure the log parser, you have to provide the Tier Name and Environment Name.

Once you have provided the tier name and env name, you have to click on the ADD button to add the tier name and env name into the table. You can also delete a configuration by selecting the checkbox corresponding to the Env name and then by clicking the Delete Selected Indices button.

Log Source

The log source is used to identify the source of the log data which you want to analyze such as: Application Logs, Security Logs, Server Logs etc. A log source is used to determine the format of the logs such as: Plain Text, JSON, XML or other structured formats. In log source, you can see the following details which are listed below:

Source: This refers to the origin or location from which the log data is being collected for further analysis. In source, you can either select: File or JournaID.
Type: This refers to the field or attribute which are used for categorizing and differentiating log events based on their characteristics, example: journaID.
Technology: This refers to the framework or technology which are used for implementing the log parser process such as: Apache_airflowlogs.
Path: This refers to the file path or location where the log files are stored and from which log data needs to be collected and parsed.
Parser: This refers to the component or module responsible for interpreting and extracting structured information from raw log data.

You can choose the type of the parser as per the technology type you have selected from their respective drop-down.All the above fields are mandatory.

Advance

To apply the log source advance settings which are optional, you have to click on the Advance button.By clicking on the advance button, you can apply the following advance settings which are listed below:

Sincedb Path: This refers to the plugin which keeps track of the current position in each file by recording it in a separate file which is named as "sincedb" i.e.: which time and rate data's index is being ingested in NFDB.
Pattern: This refers to a predefined regular expression or template that is used to identify and extract special fields or structured data from unstructured log entries such as: ^(\d{3}|\d{2}|(..\d{2})|(.\d{1})))
Negate: This refers to a parameter or option that allows you to specify whether to negate certain log entries from being processed or included in the output. You can either select True or False from the drop-down.
What: This refers to which log is to be taken to determine the new log base on the pattern.You can either select Previous or Next option from the drop-down.
Start Position:This refers to the initial point or location within the log file from which a log parser should begin reading and processing the log entries. You can either select Beginning or End by clicking on their respective radio buttons.

Once you have provided all the information in their respective fields, you can click on Next button to enter the information for Log Server.

You can also add more log sources, by clicking on the ADD button. To remove the log source window, you have to click on the Close icon.

Log Server

The log server refers to a remote server or service responsible for receiving, storing and managing log data collected from various sources. Log server centralizes and organizes the log data for analyzing, troubleshooting, and monitoring purposes. In the log server, you have the following details which are listed below:

Index Prefix: This refers to the configuration settings that allows you to specify the prefix for the index name that NF Agent will create in NFDB when it sends data to NFDB. Example: attr_prod.
Log Type: This refers to the field or attribute that is used to classify logs based on their content. Example: attr_prod.
Output Host:Port: This refers to the configuration settings that specifies the destination (listing) port for NFDB. You can also provide multiple hosts and ports. Example: ["host:port"].

Advance

To apply the log server advance settings which are optional, you have to click on the Advance button. On clicking the advance button, you apply the following advance settings which are listed below:

ssl: This is used to establish a secure connection between the NF Agent and NFDB.
ssl_certificate_verification: This is used to check whether the certificate is signed by a trusted Certificate Authority (CA) and whether it has not expired.

The index template in nfdb defines settings and mappings for indices that match a specific pattern. By managing the template, nf-agent can ensure that the indices it creates or uses adhere to the desired schema and settings, then you have to select the Manage Template checkbox

Once you have provided all the information in their respective fields, you can click on NEXT button to enter the information for Preview. You can also add more log sources, by clicking on the ADD button. To remove the log source window, you have to click on the Close icon.

Preview

In preview, you can view as well as edit all the configuration files for the logs such as: Input-Conf, Filter_nfagent.conf, Filter_error.conf, Filter_debug.conf and Output-Conf. In preview, you can add extra fields for the logs files. In preview, you can edit the following:

Input-Conf: This refers to the input plugins which enables a specific source of events to be read by NF-agent. You can also edit the input configuration by clicking on the Edit Log-Source button, once you have edited the log source, you have to click on the Save Log Source button to save the changes.
Output-Conf: This refers to the output plugins which send events data to a particular destination. You can also edit the log file by clicking on the Edit Log Server button.
Filter: This refers to filtering of the raw data from application and ingesting it into NFDB. You can also edit or delete a configuration file. You can also edit the filter by clicking on the Edit Filter button

Once you have entered all the changes to apply the changes to the DB, you have to click on the Save button and To delete a config file, you have to click on the Delete button.