Log Monitoring | Ocient Documentation

Contact Ocient® Support for assistance before altering the system logging configuration in the rolehostd.conf file.

The Ocient Hyperscale Data Warehouse™ logs events using these severities:

Error
Warn
Info
Verbose
Debug
Extended Debug

By default, the data warehouse creates the text log file /var/opt/ocient/rolehostd.log and logs events at the Info level and above.

Database logging supports extensive customization and integration with third-party tools. You can customize using the rolehostd config file. The database can return events in text, JSON, or Graylog® Extended Log Format (GELF) formats, or the database can return events in a file, to standard output (stdout), or to a UDP port. You can customize based on the log source.

Log Formats

The default log format is text. These are some example log entries.

[2020-06-09T18:11:25.701594][ INFO][role001] roleHost: All roles started
[2020-06-09T18:11:25.701881][ INFO][role001] roleHost: Starting all endpoints
[2020-06-09T18:11:25.701925][ INFO][role001] roleHost: Starting endpoint 127.0.0.1:17900
[2020-06-09T18:11:25.704905][ INFO][role001] roleHost: Starting endpoint 127.0.0.1:17600
[2020-06-09T18:11:25.708437][ INFO][tcpU008] roleHost: All endpoints started
[2020-06-09T18:11:25.709236][ INFO][role001] host: ------------------------------
[2020-06-09T18:11:25.709286][ INFO][role001] host: Rolehostd successfully started
[2020-06-09T18:11:25.709305][ INFO][role001] host: ------------------------------

Each log line contains the time, severity, thread, and the source of the log (for example, roleHost or host).

JSON-formatted logs contain the same information but in a structured format.

{"sys":"oc1.admin0", "timestamp":1591726285.701594684, "thread":"role001", "lvl":" INFO", "src":"roleHost", "msg":"All roles started"}
{"sys":"oc1.admin0", "timestamp":1591726285.701881144, "thread":"role001", "lvl":" INFO", "src":"roleHost", "msg":"Starting all endpoints"}
{"sys":"oc1.admin0", "timestamp":1591726285.701925519, "thread":"role001", "lvl":" INFO", "src":"roleHost", "msg":"Starting endpoint 127.0.0.1:17900"}
{"sys":"oc1.admin0", "timestamp":1591726285.704905192, "thread":"role001", "lvl":" INFO", "src":"roleHost", "msg":"Starting endpoint 127.0.0.1:17600"}
{"sys":"oc1.admin0", "timestamp":1591726285.708437413, "thread":"tcpU008", "lvl":" INFO", "src":"roleHost", "msg":"All endpoints started"}
{"sys":"oc1.admin0", "timestamp":1591726285.709236006, "thread":"role001", "lvl":" INFO", "src":"host", "msg":"------------------------------"}
{"sys":"oc1.admin0", "timestamp":1591726285.709286683, "thread":"role001", "lvl":" INFO", "src":"host", "msg":"Rolehostd successfully started"}
{"sys":"oc1.admin0", "timestamp":1591726285.709305246, "thread":"role001", "lvl":" INFO", "src":"host", "msg":"------------------------------"}

GELF-formatted log entries follow the format defined on the GELF via UDP page. This is an example GELF format.

{"host":"oc1.admin0", "version": "1.1", "timestamp":1591726285.701594684, "_thread":"role001", "level":"5", "_source":"roleHost", "short_message":"All roles started"}
{"host":"oc1.admin0", "version": "1.1", "timestamp":1591726285.701881144, "_thread":"role001", "level":"5", "_source":"roleHost", "short_message":"Starting all endpoints"}
{"host":"oc1.admin0", "version": "1.1", "timestamp":1591726285.701925519, "_thread":"role001", "level":"5", "_source":"roleHost", "short_message":"Starting endpoint 127.0.0.1:17900"}
{"host":"oc1.admin0", "version": "1.1", "timestamp":1591726285.704905192, "_thread":"role001", "level":"5", "_source":"roleHost", "short_message":"Starting endpoint 127.0.0.1:17600"}
{"host":"oc1.admin0", "version": "1.1", "timestamp":1591726285.708437413, "_thread":"tcpU008", "level":"5", "_source":"roleHost", "short_message":"All endpoints started"}
{"host":"oc1.admin0", "version": "1.1", "timestamp":1591726285.709236006, "_thread":"role001", "level":"5", "_source":"host", "short_message":"------------------------------"}
{"host":"oc1.admin0", "version": "1.1", "timestamp":1591726285.709286683, "_thread":"role001", "level":"5", "_source":"host", "short_message":"Rolehostd successfully started"}
{"host":"oc1.admin0", "version": "1.1", "timestamp":1591726285.709305246, "_thread":"role001", "level":"5", "_source":"host", "short_message":"------------------------------"}

The database always returns message content in the short_message field of the GELF format.

This table maps Ocient severities to GELF system log (syslog) severities.

Ocient Severity	Syslog Severity	Syslog Numeric Value
`Error`	`Error`	3
`Warn`	`Warning`	4
`Info`	`Notice`	5
`Verbose`	`Info`	6
`Debug` and `Extended Debug`	`Debug`	7

Log Appenders

A log appender is the destination for log entries. The database can send log entries to a file, a UDP port, or to stdout.

When logs are sent to the file appender, they grow continuously unless you use the truncate option in the configuration. You can use tools such as logrotate to manage log files.

This is a sample logrotate configuration.

/var/opt/ocient/log/rolehostd.log {
   rotate 5
   daily
   delaycompress
   compress
   copytruncate
   missingok
   minsize 1M
   maxsize 100M
}

Test the UDP appender using the Linux® netcat command.

$ netcat -ul 9000

Log Configuration

An Ocient System supports various logging settings to meet different needs.

Set log configuration settings in rolehostd.conf, a YAML file normally found in the system /var/opt/ocient/ directory. You can change configuration settings to suit your system logging needs in this file.

For information on checking the status of nodes, see Statistics Monitoring.

If you are using an Ocient Simulator, logging behavior and directories might differ from these descriptions.

For details on how logging works in the Ocient Simulator, see Simulator Logging.

rolehostdExample

This is an abbreviated rolehostd.conf example in YAML format. Normally, this file contains other system configurations, but this tutorial focuses on the loggingConfig section, which defines logging behaviors and scopes.

loggingConfig:
    '':  # Default scope (all logs)
        appender:
        logFilePath: /path/to/rolehostd.log
    appenders:
    - appender:
        allowLevel: warn
        logFilePath: /path/to/error.log
    - appender:
        allowLevel: warn
        format: json
        logFilePath: /path/to/error.json
  
    dump:  # Dump-specific logs
        allowLevel: debug
        appender:
            logFilePath: /path/to/dump.log
  
    query:  # Query-specific logs
        allowLevel: info
        appender:
            format: json
            logFilePath: /path/to/query.json
  
    security:  # Security-specific logs
        allowLevel: info
        appender:
            format: json
            logFilePath: /path/to/security.json
    network-logs:  # Example network logging
        allowLevel: info
        appender:
            type: UDP
            server: graylog.internal
            port: 12201
            format: gelf
            compressed: true

Scope Definition

The top level of the logging configuration is the scope, which determines the corresponding log sources for each configuration setup.

Default Scope

An empty scope string ('') applies its configuration settings to all log entries the system returns.

Named Scopes

Named scopes apply to log entries where the first component matches the name. Useful scopes include:

query — Contains information on all queries and statements executed on the system, including users, time, and other details.
security — Contains information on all connections made to the system, successful or unsuccessful.
dump — Contains information on system failures.

`allowLevel`

The allowLevel parameter specifies which severity levels of logs to capture.

allowLevel: value

Severity value definitions are:

debug — Most verbose (includes all logs).
info — Includes informational messages and greater severities.
verbose — More verbose than the info severity.
warn — Includes warnings and errors only.
error — Includes only error messages.

`appender`

Appenders define the format and directory where the database sends log files. You can configure either one or many appenders for each scope.

Single Appender

For a single appender, use the appender key.

appender:
    key: value
    [ ... ]

Multiple Appenders

For multiple output destinations, use the appenders key, followed by a list of appender keys.

 appenders:
 - appender:
     key: value
     [ ... ]
 - appender:
     key: value
     [ ... ]
 [ -...]

`format`

Defines the output format of the log files of an appender.

appender:
    format: format_type

Supported format_type values are:

text — Human-readable text format (default).
json — Structured JSON format.
gelf — GELF format for integration with Graylog.

`logFilePath`

Defines the destination directory for log files of the appender. The directory value is a file path for the appender log files, for example: /path/to/rolehostd.log.

appender:
    logFilePath: directory

`type`

Determines whether the system submits error logs through stdout or to a separate server using a TCP or UDP protocol.

appender: # standard output example
    type: stdout
appender: # UDP example
    type: UDP
    server: graylog-server.example.com
    port: 12201
    chunkSize: 1024  # Optional
    compressed: true  # Optional
appender: # TCP example
    type: TCP
    server: 192.158.1.38
    port: 12201

server

Required for TCP or UDP types.

The destination DNS name or IP address where the database sends log entries.

appender:
    server: 192.158.1.38

port

Required for TCP or UDP types.

The port where the database sends log entries. The default value is 9000.

appender:
    port: 12201

chunkSize

Optional for UDP type.

The largest packet size in bytes that the system sends to the server. The database discards log entries larger than this value.

appender:
    chunkSize: 1024

compressed

Optional for UDP type.

If you set this option to true, the system compresses packets. For details about log compression, see the GELF documentation. Otherwise, the packets are uncompressed.

appender:
    compressed: true

`truncate`

If you set this option to true, the system truncates the log file when the system starts. Otherwise, the system appends new logs to the file. The default value is false.

appender:
    truncate: true

`redactions`

You can redact data in logs at the SQL statement level using the optional redactions configuration. Enabling redaction creates another version of the logs with redacted SQL text. The redactions configuration value is statement for the redaction of part of the statement.

appender:
    redactions:
    - statement # Optional

rolehostdLog Redaction Example

This abbreviated rolehostd.conf file configuration enables log redaction at the statement level for both the rolehostd.log and query.json log files.

loggingConfig:
    ? '':
    appenders:
    - appender:
        logFilePath: /var/opt/ocient/logs/rolehostd.log
    - appender:
        logFilePath: /var/opt/ocient/logs/rolehostd_redacted.log
        redactions:
        - statement
    query:
        allowLevel: info
        appenders:
        - appender:
            format: json
            logFilePath: /var/opt/ocient/logs/query.json
        - appender:
            format: json
            logFilePath: /var/opt/ocient/logs/query_redacted.json
            redactions:
            - statement

Install an Ocient System

Set Up System Monitoring with the TIG Stack and Kapacitor

**Linux®is the registered trademark of Linus Torvalds in the U.S. and other countries.

Log Formats

Log Appenders

Log Configuration

Scope Definition

allowLevel

appender

format

logFilePath

type

truncate

redactions

Related Links