Logging Configuration - Storm Streaming Server

Storm Streaming Server utilizes popular and powerful Apache Log4j2 logging library. A single XML configuration file is provided with each server package.

Sample Configuration

You can easily configure the way Storm logs most important operations and errors. To do so, please edit the config/log4j2.xml file. The default configuration file will look like this:

                        
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN" monitorInterval="30">

    <!-- Logging Properties -->
    <Properties>
        <!-- Default logging patter -->
        <Property name="LOG_PATTERN">%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1} - %m%n</Property>

        <!-- Default logging folder -->
        <Property name="APP_LOG_ROOT">${sys:storm.defaultDirectory}/logs</Property>

        <!-- Output colors for console (please check log4j2 manual for reference) -->
		<Property name="CONSOLE_WARN_COLOR">yellow</Property>
		<Property name="CONSOLE_ERROR_COLOR">red</Property>
		<Property name="CONSOLE_INFO_COLOR">white</Property>
		<Property name="CONSOLE_DEBUG_COLOR">bright_white</Property>
		<Property name="CONSOLE_TRACE_COLOR">blue</Property>

    </Properties>

    <Appenders>

        <!-- Console Appender -->
        <Console name="Console" target="SYSTEM_OUT" follow="true">
             <PatternLayout
                  disableAnsi="false"
                  pattern="%highlight{%d{HH:mm:ss} %-5p %c{1} - %m%n}{FATAL=${CONSOLE_ERROR_COLOR} blink,
                           ERROR=${CONSOLE_ERROR_COLOR}, WARN=${CONSOLE_WARN_COLOR} bold, INFO=${CONSOLE_INFO_COLOR},
                           DEBUG=${CONSOLE_DEBUG_COLOR} bold, TRACE=${CONSOLE_TRACE_COLOR}}"/>
        </Console>

        <!-- RollingFile Adapter from INFO to ERROR (skipping DEBUG & TRACE) -->
        <RollingFile name="errorLog" fileName="${APP_LOG_ROOT}/log-error.log" filePattern="${APP_LOG_ROOT}/old-erros/log-error-%d{yyyy-MM-dd}-%i.log">
            <LevelRangeFilter minLevel="DEBUG" maxLevel="ERROR" onMatch="ACCEPT" onMismatch="DENY"/>
            <PatternLayout pattern="${LOG_PATTERN}"/>
            <Policies>
                <SizeBasedTriggeringPolicy size="10MB"/>
            </Policies>
            <DefaultRolloverStrategy max="100"/>
        </RollingFile>

       <!-- RollingFile Adapter from WARN to ERROR  -->
        <RollingFile name="allLog" fileName="${APP_LOG_ROOT}/log-all.log" filePattern="${APP_LOG_ROOT}/old/log-all-%d{yyyy-MM-dd}-%i.log">
        	<LevelRangeFilter minLevel="WARN" maxLevel="ERROR" onMatch="ACCEPT" onMismatch="DENY"/>
            <PatternLayout pattern="${LOG_PATTERN}"/>
            <Policies>
                <SizeBasedTriggeringPolicy size="10MB"/>
            </Policies>
            <DefaultRolloverStrategy max="100"/>
        </RollingFile>

    </Appenders>

    <Loggers>

        <Root level="debug">
            <AppenderRef ref="Console"/>
            <AppenderRef ref="allLog"/>
            <AppenderRef ref="errorLog"/>
        </Root>

    </Loggers>

</Configuration>
                    

Configuration Explanation

For detailed explanation and configuration option please check Apache log4j2 page: https://logging.apache.org/log4j/2.x/. Below you’ll find a general overview for basic options.

1. Basic status

   The line below defines minimal logging level via status property. There are 7 levels in total: ALL, DEBUG, INFO, WARN, ERROR, FATAL, OFF. If you wish to log everything that console outputs you can replace status value with INFO. Parameter “monitorInterval” defines how often changes in the configuration file will be checked for changes (you don’t have to restart server to modify logging settings).

                           
   <Configuration status="WARN" monitorInterval="30">
                       

2. Log pattern

   Log patterns defines how each entry is formatted.

                           
   <Property name="LOG_PATTERN">%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1} - %m%n</Property>
                       

   Field explanation:

   %d{yyyy-MM-dd HH:mm:ss}	Element defines how date and time are logged.
   %-5p	Element represents logging level e.g. INFO, ERROR, WARN.
   %c{1}	Element represents class name.
   %m%n	Element represent the message and new line (so the next message is added below).

   Table 1. Log pattern fields table
3. Console adapter

   This part controls what output is visible in the console (terminal / Powershell etc.). It uses modified log pattern to include colors.

                           
   <Console name="Console" target="SYSTEM_OUT" follow="true">
       <PatternLayout
           disableAnsi="false"
           pattern="%highlight{%d{HH:mm:ss} %-5p %c{1} - %m%n}{FATAL=${CONSOLE_ERROR_COLOR} blink,
                    ERROR=${CONSOLE_ERROR_COLOR}, WARN=${CONSOLE_WARN_COLOR} bold, INFO=${CONSOLE_INFO_COLOR},
                    DEBUG=${CONSOLE_DEBUG_COLOR} bold, TRACE=${CONSOLE_TRACE_COLOR}}"/>
   </Console>
                       

4. Rolling adapter

   Rolling adapter saves logs to a specified file. Once the number of entries or file size exceeds certain values, file is renamed to include date and a new one is created in its place.

                           
   <RollingFile name="errorLog" fileName="${APP_LOG_ROOT}/log-error.log" filePattern="${APP_LOG_ROOT}/old-erros/log-error-%d{yyyy-MM-dd}-%i.log">
       <LevelRangeFilter minLevel="DEBUG" maxLevel="ERROR" onMatch="ACCEPT" onMismatch="DENY"/>
       <PatternLayout pattern="${LOG_PATTERN}"/>
       <Policies>
           <SizeBasedTriggeringPolicy size="10MB"/>
       </Policies>
       <DefaultRolloverStrategy max="100"/>
   </RollingFile>
                       

   Field explanation:

   minLevel	Only logs above this level will be saved to a file.
   maxLevel	Only logs below this level will be saved to a file.
   SizeBasedTriggeringPolicy:size	Maximum file size.
   DefaultRolloverStrategy:max	Max entries count.

   Table 2. Field explanation table