Section 1. General questions
Q 1.1: What was the motivation behind developing SEC?
What is the "history" of this tool?
Q 1.2: How many sites are using SEC?
Q 1.3: How people have used SEC so far?
Q 1.4: What about SEC performance?
How much load can it handle?
Section 2. Getting started
Q 2.1: What sort of input sources can I use with SEC?
Q 2.2: Can I use SEC for processing binary input?
Q 2.3: Does SEC have "direct interface" to some network
management platforms (like Tivoli or HP OpenView)?
Q 2.4: Does SEC support network topology databases of
HP OpenView, Tivoli, or some other platforms? How can I use network topology
information in the event correlation process?
Q 2.5: How can I integrate SEC with HP OpenView Network
Node Manager?
Q 2.6: How can I integrate SEC with HP OpenView ITO?
Section 3. Rules and event correlation operations
Q 3.1: I don't understand the difference between rules
and event correlation operations. Are they one and the same?
Q 3.2: How are event correlation operations identified
inside SEC?
Q 3.3: How can I see what event correlation operations
are currently active? How can I see the state of individual event
correlation operations? How can I see what SEC is currently doing?
Q 3.4: How does 'reset' action work?
Q 3.5: What will happen if I reset an event correlation
operation that does not exist? Is it safe to do so?
Q 3.6: I am afraid that by instructing SEC to create many
contexts and event correlation operations at runtime, I will also create
performance problems?
Q 3.7: How can I use regular expression modifiers in
SEC rule definitions (like //i, //m, etc.)?
Q 3.8: How can 'pattern' parameter of the Pair or
PairWithWindow rule influence the 'pattern2' parameter?
Q 3.9: When should I use %1, %2, ... special
variables?
Q 3.10: The SingleWithScript rule provides the external
program (given by its 'script' parameter) with all context names through
the program's standard input. What should I do to handle these contexts
efficiently?
Q 3.11: I am using the SingleWithScript rule and the
external script given by the 'script' parameter always returns non-zero
(e.g., 255). What's wrong?
Q 3.12: How does the 'continue' parameter work for
the SingleWithScript rule?
Q 3.13: How can I save context names to disk when SEC
is shut down, and read them in after SEC has been restarted?
Q 3.14: How can I add similar events to a context, and
report all these events at once after a timeout (i.e., report them when no
more such events have occurred during last N seconds)?
Q 3.15: How can I add similar events to a context, and
report all gathered events after N seconds?
Q 3.16: How can I generate immediate alert on the first
suspicious event, and provide a detailed report of all suspicious events after
a certain period of inactivity (i.e., there have been no suspicious events
for the last N seconds)?
Q 3.17: How can I keep an eye on suspicious events
for longer periods of time, and report only those kinds of events that
occur more frequently?
Q 3.18: How can I set up bi-directional
communication between SEC and its subprocess that was started by 'spawn'
action?
Q 3.19: How can I run 'shellcmd' actions in an ordered
manner (i.e., an action is not started before the previous one has
completed)?
Q 3.20: How can I force SEC to recognize the cluster
of events (say, events A, B, and C), if those events can have an arbitrary
order (A, B, C; C, A, B; B, C, A; etc.)?
Q 3.21: I have started a long-running subprocess from
SEC, but when I send SIGHUP or SIGTERM to SEC, the subprocess will not receive
SIGTERM as it should. What can be done about it?
Q 3.22: How can I load Perl modules that could be used at
SEC runtime by 'eval' actions?
Q 3.23: How can I use variables in the SEC code, e.g.,
how can I print out the content of all SEC contexts at SEC shutdown?
Q 3.24: How can I store all data related to contexts
(context names, their event stores, lifetimes, etc.) to disk and reload them
at a later time?
Q 3.25: How can I write a rule for several input sources
that would be able to report the name of the source for matching lines?
Section 4. Miscellaneous
Q 4.1: My platform does not have SIGABRT.
What signal should I use for "soft reloads"?
Q 4.2: Why are several SEC features (like 'spawn' action)
not supported on Win32 with ActivePerl?
Q 4.3: I am trying to run SEC 2.1.X on RedHat Linux, but
it does not recognize my rules at all. What's wrong?
Q 4.4: I have set up a named pipe as input file with
-input option, but SEC is unable to open it, although SEC has a permission
to read it. Why doesn't my setup work?
A: In summer 1999 I had to solve a logfile monitoring task that involved event correlation. Since I could not crack this problem with standard HP OpenView ITO installation (without ECS), I started to look for a free event correlation system, but for my surprise was unable to locate any such software. The closest match was logsurfer, but this tool did not satisfy my needs entirely. To solve the problem I had, I wrote a very simple and robust tool that could correlate events received from a logfile. This tool was rather primitive - there was no support for contexts, many ruletypes that SEC has were not implemented, etc. In summer 2000 I had to get back to the issue of event correlation, and since meanwhile no freeware event correlation tools had become available on Internet, I decided to rewrite the old code from 1999. After extensive testing, SEC-1.0 was finally released in March 23, 2001.
Unfortunately, the issue of price is rather painful in the area of network management and event correlation. All commercial event correlation engines are quite expensive, and SEC tries to be a cost-effective alternative to them. Though commercial engines are very powerful, allowing you to solve very complex event correlation problems, many companies and organizations never face so complicated event correlation issues, and therefore don't actually need that powerful and that expensive software. The motivation behind SEC was to create a freeware and platform independent lightweight tool for event correlation, that could be used for both local and central event correlation. SEC is distributed under the terms of GNU GPL.
Q 1.2: How many sites are using SEC?
A: It is very difficult to tell the exact figure, because no registration is
required to download it :-) However, the SEC project home page at SourceForge
has some download statistics available
(see
http://www.sourceforge.net/projects/simple-evcorr/).
Q 1.3: How people have used SEC so far?
A: SEC has been successfully applied as a central correlation engine for
HP OpenView NNM and ITO (both on HP-UX and Solaris). It has been also used
as a local correlation engine for HP OpenView ITO agents (on Linux, HP-UX,
and Solaris). Quite many people have used it with Snort.SEC has also been
applied as a logfile monitoring tool on central loghosts, etc., etc.
Q 1.4: What about SEC performance? How much load can it handle?
A: It is pretty capable. I have used SEC on an old non-MMX 200MHz Pentium
(performing only at 80 bogomips) for almost a year, and it was able to
handle the load of 100-200 events per second for several hours every day.
There were more than 60 rules in the configuration file, many of them with
complex regular expressions as search patterns. To be frank, I was quite
surprised myself how well the tool handled this load on such a weak
hardware. Here the authors of perl deserve full credit for inventing so
efficient language and for writing so capable interpreter :)
A: You can use any input source, as long as it takes the form of a file,
named pipe, or standard input. Events from the input source can have any
form, as long as they can be matched by perl-style regular expressions.
This means that SEC can be used for correlating events that are written
to a logfile. Also, SEC can be used as a postprocessing system for an
arbitrary application that is able to write its output to a named pipe or
standard output.
Q 2.2: Can I use SEC for processing binary input?
A: In principle, you can, because perl regular expressions work on binary
input. Since SEC reads lines with read(2), one byte at a time, it will not be
confused by binary data. However, you have to take into account that the
bytes are organized into lines after they are read, by considering \n
(or ASCII 10) bytes as line terminators. By using RegExpN (N=1,2,..) pattern
type, you can recognize patterns in the binary data.
Q 2.3: Does SEC have "direct interface" to some network management
platforms (like Tivoli or HP OpenView)?
A: One of the design ideas behind SEC was modularity and platform
independence - SEC should not be tied to some particular application or
system. Therefore, SEC uses file interface and regular expressions that
should be general enough for any application. If integration with some
particular application is desired that is unable to direct its output to
a file-like destination, an event converter needs to be written (sample
event converter for HP OpenView ITO/VPO is included in the SEC package).
Commercial event correlation tools are often heavyweight and monolithic -
such packages try to include support for as many event formats as possible.
SEC takes a different approach, trying to be as lightweight and modular
as possible.
Q 2.4: Does SEC support network topology databases of HP OpenView, Tivoli,
or some other platforms? How can I use network topology information in the
event correlation process?
A: There is no support for any specific network topology database format
in the SEC core (see previous question). However, SEC allows you to integrate
custom scripts and programs into SEC event flow (see SingleWithScript rule
and 'spawn' action in SEC man page). Basically, you have to write a script or
a program that is able to query the topology database you have, and then
integrate this program with your SEC rule base.
Q 2.5: How can I integrate SEC with HP OpenView Network Node Manager?
A: Network Node Manager 5.0 and previous releases write all the events they
see to trapd.log file. Therefore, you just have to specify this file as input
for SEC. Starting from version 6.0, Network Node Manager does no longer
produce trapd.log file by default. To force it to do so, you have to edit
pmd.lrf file, adding -SOV_EVENT;t option to the file:
OVs_YES_START::-SOV_EVENT;t:OVs_WELL_BEHAVED:15:PAUSE
After that, execute following commands:
ovaddobj pmd.lrf
For producing output events from SEC, you can use Network Node Manager's
ovevent utility. For detailed information, see ov_event(5), lrf(4), and
ovevent(1) manual pages.
Q 2.6: How can I integrate SEC with HP OpenView ITO?
A: Use itostream plugin that is part of the SEC package. The plugin has
been tested with ITO5.3 and ITO6.0, and has been found working
with HP-UX and Solaris management servers, but also with HP-UX, Solaris,
and Linux agents.
To use the plugin, you first need to compile it. The compiling
instructions are located in the itostream.c file, but if you are compiling
on management server, the following line should be sufficient:
gcc -o itostream itostream.c -L/opt/OV/lib -lopcsv -lnsp
On agents use -DAGENT flag, e.g.,
gcc -o itostream itostream.c -DAGENT -L/opt/OV/lib -lopc -lnsp
On some agent platforms the /opt/OV/lib directory is not included in the
shared library search path, which results an error message when you try to
run itostream binary. To include /opt/OV/lib in search path, use
-Xlinker -rpath options:
gcc -o itostream itostream.c -DAGENT -L/opt/OV/lib -lopc -lnsp \
-Xlinker -rpath /opt/OV/lib
Also, some ITO agent platforms don't have /opt/OV/lib/libopc.*
library, which is normally just a symbolic link to
/opt/OV/lib/libopc_r.* library. In that case try
to use the following commandline:
gcc -o itostream itostream.c -DAGENT -L/opt/OV/lib -lopc_r -lnsp
(i.e., use -lopc_r option instead of -lopc).
In order to use itostream binary on the management server, you need to
enable output for external MSI plugins. To do that, open the ITO GUI
Node Bank, and go to Actions->Server->Configure. Then check "Enable Output"
option, and close the window.
If you wish to use itostream on particular ITO managed node, right-click
on the managed node icon, and go to Modify->Advanced Options. Then check
"Enable Output" option, and close the window.
Itostream takes 2 parameters: the name of MSI interface (you can use
arbitrary string here, like "test" or "mymsi"), and timeout N -
when itostream has seen no data for the last N seconds, it will try to
reconnect to the local ITO agent or ITO management server. After
startup, itostream will write ITO messages to its standard output, one
message per line. Itostream's standard output can be directed to a pipe or
file, which can be input for SEC. Here are some sample lines from itostream
output:
time=1025713202 sev=normal node=server1.mydomain app=TEST obj=TEST msg_grp=Network msg_text=node up
A: No, they are not. Basically, rules are instructions to SEC that tell
which event correlation operations to start and when. Rules are static in
their nature - their number will not change at runtime, unless you have
updated the configuration file(s) and sent SIGHUP or SIGABRT to SEC.
Event correlation operations, in turn, are dynamic - they are started by the
rules and they terminate after their job is done.
There is no 1-1 relationship between the rules and event correlation
operations - there can be many running event correlation operations that
were all started by the same rule.
The task of event correlation operation is to detect a composite event.
After the rule has detected an event that could start a composite event,
event correlation operation is started to check whether the composite event
will indeed occur.
Say that composite event C is defined as follows:
C = {3 events A will occur within the time window of 60 seconds}
Suppose that you have written SingleWithThreshold rule for detecting C.
After the rule has detected event A, it will start an event correlation
operation that will be active for 60 seconds. If this operation sees two
more A events, it will execute the specified action.
It should also be noted that Single, SingleWithScript, Suppress, and Calendar
rules never start event correlation operations, since they don't involve
the matching of several events over a certain time window (i.e., their design
allows them to complete their work almost immediately without starting an
event correlation operation).
Q 3.2: How are event correlation operations identified inside SEC?
A: After the rule has started an event correlation operation, this event
correlation operation needs to be distinguished from other running operations.
To do this, SEC assigns a key to the operation, that is composed from
configuration file name, rule ID, and description string of the composite
event.
Say that you have configuration file my.conf with one rule in it:
type=SingleWithThreshold
Suppose that SEC observes input line "user admin login failure on tty1".
This matches pattern 'user (\S+) login failure on (\S+)', and SEC will
now construct event description string for the composite event. After
replacing $1 and $2 with actual values, event description string will become
"Repeated login failures for user admin on tty1".
Before starting an event correlation operation, SEC will calculate a key for
that operation. The key is calculated from configuration file name, rule ID,
and event description string of the composite event, yielding:
my.conf | 0 | Repeated login failures for user admin on tty1
Since the rule was the first one in the configuration file, its ID is 0.
The ID for the second rule would be 1, for the third rule 2, etc.
When SEC observes input line "user USERNAME login failure on TERM", it
will first calculate the key and check if there already is an event
correlation operation with that key. If such operation exists, detected line
will be correlated by the existing operation. Otherwise, new event
correlation operation will be started.
This algorithm means that by using appropriate event description strings,
you can change the essence of event correlation.
For instance, if you use 'Repeated login failures for user $1' for event
description string, you will just count login failures for different users,
disregarding terminal name. Therefore, following three lines will be
correlated by the same event correlation operation:
user admin login failure on tty1
However, if you use 'Repeated login failures for user $1 on $2' for event
description string, the three lines above will each start a separate event
correlation operation.
Since the configuration file and rule ID are present in the keys, event
correlation operations started by different rules will not clash, even if
their event description strings are identical.
Q 3.3: How can I see what event correlation operations
are currently active? How can I see the state of individual event
correlation operations? How can I see what SEC is currently doing?
A: Send SIGUSR1 signal to the SEC process, this will cause SEC to dump
detailed information about its state to the dumpfile (given with -dump
option). The information includes details about event correlation operations
and contexts that are currently active, SEC performance and rule usage
statistics, etc.
Q 3.4: How does 'reset' action work?
A: Suppose you have two rules in your configuration file my.conf:
type=Single
type=SingleWithThreshold
Say that SEC will observe the following lines:
user admin login failure on tty1
The second rule will start a separate event correlation operation for each
of the lines. The keys of these operations are:
my.conf | 1 | Repeated login failures for user admin on tty1
(since the rule was the second one in the configuration file, its ID is 1).
When SEC will observe the line 'user admin logged in on tty5', it will
evaluate $1 and $2 variables, yielding the following action:
reset +1 Repeated login failures for user admin on tty5
This means that SEC has to terminate the operation which has been started
by the next rule (+1) and which has the key
my.conf | 1 | Repeated login failures for user admin on tty5
Terminating the event correlation means that SEC will immediately delete all
internal data structures related to the operation, and no actions or other
activities will be performed.
Since the terms "next to the first" and "second" are identical,
another way to write the same action is:
action=reset 2 Repeated login failures for user $1 on $2
If there is no rule number specified in the action definition, e.g.
action=reset Repeated login failures for user $1 on $2
then SEC will assume the rule number to be a wildcard, constructing all
possible keys and trying to find and delete corresponding operations.
If there are 5 rules in the configuration file, SEC would look for the
operations with the following keys:
my.conf | 0 | Repeated login failures for user admin on tty5
If some (or all) of the operations do not exist, no deletion will be
performed for the missing operations.
Q 3.5: What will happen if I reset an event correlation operation that
does not exist? Is it safe to do so?
A: Yes, it is safe. Event correlation operations are implemented as a perl
hash inside SEC, and deleting an element from a hash is always a safe
operation in perl, even if there is no element for a given key in a hash.
Q 3.6: I am afraid that by instructing SEC to create many contexts and
event correlation operations at runtime, I will also create performance
problems?
A: Contexts, event correlation operations, and most other SEC entities are
organized into perl hashes inside SEC. Perl hash is a sophisticated data
type that can be searched very quickly with very little overhead. This means
that SEC can operate on contexts, event correlation operations, and other
SEC entities efficiently with minimal consumed CPU time.
Q 3.7: How can I use regular expression modifiers in SEC rule definitions
(like //i, //m, etc.)?
A: SEC regular expression definitions don't include surrounding slashes, and
therefore it looks like there is no place for modifiers. Fortunately, perl
regular expressions allow you to use modifiers inside regular expressions:
/your_regexp/i can be expressed as /(?i)your_regexp/
Therefore, if you would like to set 'pattern' field to /[A-Z]/i,
correct way of doing that would be 'pattern=(?i)[A-Z]'.
Q 3.8: How can 'pattern' parameter of the Pair or PairWithWindow rule
influence the 'pattern2' parameter?
A: If 'pattern' parameter is a regular expression, you can use $1, $2, ...
special variables in the 'pattern2' parameter. E.g., if
ptype=regexp
and event 'interface 192.168.1.1 down' is observed, $1 will be set to
'192.168.1.1' and 'pattern2' will become 'interface (192\.168\.1\.1) up'.
However, the most efficient way would be to write
ptype=regexp
since this will instruct SEC to use substring search for detecting the
second pattern, which is faster than regular expression matching.
NOTE that if 'ptype2' is set to 'regexp', all regexp special symbols are
quoted before replacing $1 with its actual value in 'pattern2', in order to
prevent unwanted side effects.
If 'ptype2' is set to 'substr', no such quoting will be done,
of course, and the value of $1 is left as is before replacement.
You can see the 'pattern2' values of all pending Pair* event correlation
operations in the SEC dumpfile, when you send SIGUSR1 to the SEC process.
Q 3.9: When should I use %1, %2, ... special variables?
A: These special variables can be used in Pair and PairWithWindow rules with
both 'ptype' and 'ptype2' set to 'regexp'. In that case, $1, $2, ... variables
will be set to backreference values of 'pattern2', and %1, %2, ... can be
used to retreive the backreference values of 'pattern'. E.g., if
ptype=regexp
and event 'interface 192.168.1.1 down name=Serial1' will be observed,
followed by 'interface 192.168.1.1 up', then 'desc2' parameter will be set
to 'interface 192.168.1.1 Serial1 up'.
Q 3.10: The SingleWithScript rule provides the external program (given
by its 'script' parameter) with all context names through the program's
standard input. What should I do to handle these contexts efficiently inside
the program?
A: The most important thing would be to organize the context names so that
they could be quickly searched by the program (unless the program doesn't
really need to know the context names to carry out the processing). For
instance, if the program is written in perl, you can use hashes to organize
context names:
while (<STDIN>) { chomp; $context{$_} = 1; }
This code fragment creates a hash with keys corresponding to the context
names. To check whether the context exists, use perl exists() operator
(e.g., if (exists($context{CONT})) { ... }).
Q 3.11: I am using the SingleWithScript rule and the
external script given by the 'script' parameter always returns non-zero
(e.g., 255). What's wrong?
A: This is caused by a bug in the perl's close() function that is present on
some platforms. The code of SEC-2.1.4 contains workaround for the bug.
If you have SEC-2.1.3 or lower, check whether your script handles its
standard input, and if it doesn't, change it accordingly.
The SingleWithScript rule feeds the names of all currently existing contexts
to the standard input of the script. If the script does not read them, the
perl's close() function reports to the SEC process that the script terminated
with an error. Therefore, if you wish to ignore the context names inside your
script, add 'cat >/dev/null' (or something similar) to your script, or use
'cat >/dev/null && /mypath/myprog' for the value of the 'script' parameter.
Q 3.12: How does the 'continue' parameter work for the SingleWithScript
rule?
A: Since the runtime of the script that is specified by the 'script'
parameter is not limited in any way, waiting for that script to
complete could freeze the entire event processing for an indefinite amount
of time. Therefore, once the 'pattern' parameter has matched
an input line, SEC immediately looks at the 'continue' parameter, and
passes the input line to the next rule, if 'TakeNext' was specified as a
value.
Q 3.13: How can I save context names to disk when SEC is shut down,
and read them in after SEC has been restarted?
A: Add -intevents option to SEC commandline, and write rules for saving
context names when SEC_SHUTDOWN event is observed and for reading context
names in when SEC_STARTUP event is observed. For writing context names into
a file, use SingleWithScript rule:
# save context names
type=SingleWithScript
# read in context names, prepending 'SEC_CONTEXT: ' prefix to every name
type=Single
# Create contexts, based on the information received from the previous rule
type=Single
Q 3.14: How can I add similar events to a context, and
report all these events at once after a timeout (i.e., report them when
no more such events have occurred during last N seconds)?
A: If you want to gather and report logfile lines
of ftp sessions, for instance, consider the following rule:
# Add the logfile line to its session context ftp_PID (if the context
type=single
Q 3.15: How can I add similar events to a context, and report all gathered
events after N seconds?
A: Suppose you want to gather and report logfile lines of ftp sessions.
Consider the following rules:
# If there is no context for the ftp session at the moment (e.g.,
type=single
# Add logfile line to its session context
type=single
Q 3.16: How can I generate immediate alert on the first suspicious event,
and provide a detailed report of all suspicious events after a certain period
of inactivity (i.e., there have been no suspicious events for the last N
seconds)?
A: Suppose you want to receive an alert when ftp session is started from
the suspicious host, and you wish to receive a detailed report
of the session when the host has been inactive for 20 minutes:
# This rule will set up a context ftp_PID that is used for saving
type=Single
# This rule will save each session event to its session context ftp_PID,
type=Single
Q 3.17: How can I keep an eye on suspicious events for longer periods of
time, and report only those kinds of events that occur more frequently?
A: As an example, see the following ruleset for "BAD LOGIN FROM host"
events:
# When "BAD LOGIN FROM host" is seen for the first time, contexts
type=Single
# Save "BAD LOGIN FROM host" event to its context, and expand the
type=Single
Q 3.18: How can I set up bi-directional communication between SEC and
its subprocess that was started by 'spawn' action?
A: When another process is started with 'spawn' from SEC,
it can send data to SEC by writing to standard output (internally, the
standard output of the process is redirected to a pipe that SEC reads).
To send data from SEC to the spawned process, set up a named pipe or file
from the process and use actions like 'write', 'report', etc. to write to
that pipe or file.
Q 3.19: How can I run 'shellcmd' actions in an ordered manner (i.e.,
an action is not started before the previous one has completed)?
A: Suppose you have the following rule definition:
type=Calendar
Since the runtime of the 'shellcmd' actions is not limited in any way,
SEC starts the actions as separate background processes, in order not
to freeze the whole event processing. Although the first action
(cat /tmp/myreport | mail root@localhost) is started before the second one
(rm -f /tmp/myreport), it is NOT guaranteed (neither by SEC nor by OS) that
the first action has already terminated by the time the second
action starts. Furthermore, since the commandlines are first processed
by the shell, it could well happen that the second action is
actually executed first, especially if its commandline is much simpler and
takes less time to process.
Therefore, the rule definition above often produces an empty e-mail
message, since the file is removed just before 'cat' gets to it.
In order to avoid such unwanted behaviour, you could use single 'shellcmd'
action together with single commandline, and take advantage of the shell's &&
control operator:
type=Calendar
i.e., the file /tmp/myreport is not removed before the 'mail' command has
completed successfully. Another way to solve this problem is to put all
your commands into a separate shell script, and give the name of the script
to 'shellcmd' action as a parameter.
Q 3.20: How can I force SEC to recognize the cluster
of events (say, events A, B, and C), if those events can have an arbitrary
order (A, B, C; C, A, B; B, C, A; etc.)?
A: Determine the size of the cluster in time (e.g., all those events must
be observed within 300 seconds for the cluster to be
complete), and use Single rules in the following manner:
type=single
type=single
type=single
type=single
These rules set up contexts for events A, B, and C, and generate a metaevent
CHECK_FOR_CLUSTER that will force the last rule to generate an
output event "Cluster detected!", if all contexts exist at a given moment.
The existence of contexts means that A, B, and C have all been observed
within the last 300 seconds (in arbitrary order).
Note that since the SEC context expressions can contain not only logical
AND-operators but OR- and NOT-operator as well together with parantheses, the
definition of "event cluster" can also contain choices and negative conditions
(e.g., the cluster is considered to be complete when event A has been observed,
either B or C or both have been observed, and D has not been observed).
Q 3.21: I have started a long-running subprocess from SEC, but when I
send SIGHUP or SIGTERM to SEC, the subprocess will not receive SIGTERM as it
should. What can be done about it?
A: When a command is started from a perl script with system() or open() call,
perl checks whether the command contains shell metacharacters, and if
it does, perl will not call the command directly but through the shell
that will interpret your commandline first.
When your platform is UNIX, this normally means that
/bin/sh -c your_command
is forked first from SEC, and after the command has been interpreted by the
shell, another process will forked from the shell for your_command.
This means that when SEC is sending SIGTERM to its child processes,
your_command will NOT receive SIGTERM, but it will be sent to the shell
that started it.
In order to avoid such unwanted behaviour (and save one slot
in your process table), use shell's exec builtin command.
When exec is prependend to your commandline, the shell will not
fork a separate process for your command, but it will be executed
inside the current process. E.g., when you specify
action=spawn exec /usr/local/bin/myscript.pl 2>/var/log/myscript.log
an extra process will not be created for myscript.pl, although the
commandline contains the shell redirection metacharacter '>'.
Q 3.22: How can I load Perl modules that could be used at SEC runtime by
'eval' actions?
A: Add -intevents option to SEC commandline, and write a rule for loading
the necessary module when SEC_STARTUP event is observed. The following
rule will load the SNMP module, and terminate SEC if the loading failed:
type=single
Note that before attempting to load the SNMP module, the %a variable will be
set to zero, since if the SEC 'eval' action fails, it will not change the
previous value of its variable. Therefore, if %a is still zero after the load
attempt, the attempt was not successful.
Q 3.23: How can I use variables in the SEC code, e.g., how can I print
out the content of all SEC contexts at SEC shutdown?
A: Using variables in the SEC code allows one to add new features to SEC and
change the SEC behavior at runtime; however, it is highly recommended to study
the code thoroughly before writing rules that access the variables. Note that
it will make sense to access SEC global variables only, since local variables
are (generally) not visible when you call the 'eval' action. In order to learn,
which global variables are available for tweaking, study the first section of
the SEC code that contains definitions of global variables. Then study the rest
of the code thoroughly how the variable of interest is used, in order to get
a good picture how to manipulate the variable safely. Since the execution of
the 'eval' action is taking place in the main::SEC name space, you have to use
the main:: prefix for accessing variables in the SEC code.
The following example reads the hash of SEC contexts (%main::context_list),
and prints out the content of all SEC contexts at SEC shutdown:
type=Single
Q 3.24: How can I store all data related to contexts (context names, their
event stores, lifetimes, etc.) to disk and reload them at a later time?
A: Use the perl Storable module for saving and restoring the hash of SEC
contexts (%main::context_list). The following actions save and restore
the context hash, using the file CONTEXTS:
action=eval %o ( store(\%main::context_list, "CONTEXTS") )
action=eval %o ( use Storable; %main::context_list = %{retrieve("CONTEXTS")} )
Q 3.25: How can I write a rule for several input sources
that would be able to report the name of the source for matching lines?
A: Use the PerlFunc pattern type that has the input source name as one of its
input parameters, and return the input source name from the pattern function.
E.g., the following rule matches the "File system full" messages with
a regular expression, and sets $1 and $2 variables to the file system and input
source names:
type=single
If the "/opt: file system full" message is logged to /var/log/server1.log,
the rule writes "File system /opt full (/var/log/server1.log)" to standard
output.
A: Many platforms that do not have SIGABRT define SIGIOT instead. On such
platforms, you can use SIGIOT in place of SIGABRT.
Q 4.2: Why are several SEC features (like 'spawn' action) not supported
on Win32 with ActivePerl?
A: Unfortunately, in ActivePerl several Perl system functions are not
implemented properly, and ActivePerl also makes very little effort to emulate
system functions that are not present on Win32 platform. Instead of using
SEC with ActivePerl, run SEC with CygWin Perl, which provides much better
implementation of Perl system functions.
Section 2. Getting started
Q 2.1: What sort of input sources can I use with SEC?
ovstop pmd
ovstart pmd
time=1025713224 sev=major node=server2.mydomain app=opsystem obj=disk msg_grp=OS msg_text=Disk fault
time=1025713227 sev=critical node=server2.mydomain app=opsystem obj=server msg_grp=OS msg_text=node down
Section 3. Rules and event correlation operations
Q 3.1: I don't understand the difference between rules and event
correlation operations. Are they one and the same?
ptype=RegExp
pattern=user (\S+) login failure on (\S+)
desc=Repeated login failures for user $1 on $2
action=shellcmd notify.sh "%s"
window=60
thresh=3
user admin login failure on tty5
user admin login failure on tty2
ptype=RegExp
pattern=user (\S+) logged in on (\S+)
desc=User $1 successful login
action=reset +1 Repeated login failures for user $1 on $2
ptype=RegExp
pattern=user (\S+) login failure on (\S+)
desc=Repeated login failures for user $1 on $2
action=shellcmd notify.sh "%s"
window=60
thresh=3
user admin login failure on tty5
user admin login failure on tty2
my.conf | 1 | Repeated login failures for user admin on tty5
my.conf | 1 | Repeated login failures for user admin on tty2
my.conf | 1 | Repeated login failures for user admin on tty5
my.conf | 2 | Repeated login failures for user admin on tty5
my.conf | 3 | Repeated login failures for user admin on tty5
my.conf | 4 | Repeated login failures for user admin on tty5
/your_regexp/m can be expressed as /(?m)your_regexp/
etc.
(see perlre(1) man page)
pattern=interface (\S+) down
ptype2=regexp
pattern2=interface ($1) up
pattern=interface (\S+) down
ptype2=substr
pattern2=interface $1 up
pattern=interface (\S+) down name=(\S+)
ptype2=regexp
pattern2=interface ($1) up
desc2=interface $1 %2 up
ptype=SubStr
pattern=SEC_SHUTDOWN
context=SEC_INTERNAL_EVENT
script=cat > /tmp/sec_contexts.dump
desc=Saving the SEC contexts
action=none
ptype=SubStr
pattern=SEC_STARTUP
context=SEC_INTERNAL_EVENT
desc=Read in previously saved SEC contexts
action=spawn perl -ne 'print "SEC_CONTEXT: $_"' /tmp/sec_contexts.dump
ptype=RegExp
pattern=^SEC_CONTEXT: (.*)
desc=Recreate context $1
action=create $1
# does not exist, 'add' action will create it), and expand the context
# lifetime for the next 30 minutes. When no more matching lines have
# been seen during the last 30 minutes, all events that were previously
# saved to the context will be mailed to root@localhost
ptype=regexp
pattern=ftpd\[(\d+)\]
desc=event of ftp session $1
action=add ftp_$1 $0; set ftp_$1 1800 (report ftp_$1 /bin/mail root@localhost)
# the session was just opened, or the session context existed before
# but expired due to long session), create it. Note that the following
# logfile lines will not match the rule as long as the context ftp_PID
# exists. The context will exist for 30 minutes, and when it expires,
# all events that were previously saved to the context will be mailed
# to root@localhost. If the ftp session is still active after 30 minutes,
# the next logfile line from that session will create the context again.
continue=takenext
ptype=regexp
pattern=ftpd\[(\d+)\]
context=!ftp_$1
desc=create context for ftp session $1
action=create ftp_$1 1800 (report ftp_$1 /bin/mail root@localhost)
ptype=regexp
pattern=ftpd\[(\d+)\]
context=ftp_$1
desc=event of ftp session $1
action=add ftp_$1 $0
# the session data. The rule will also send an immediate alert that
# a session from the suspicious host was started
ptype=RegExp
pattern=ftpd\[(\d+)\]: FTP LOGIN FROM suspicious-host.domain.com
context=!ftp_$1
continue=TakeNext
desc=FTP session from suspicious-host.domain.com
action=create ftp_$1; pipe '%s' /bin/mail root@localhost
# and expand the context lifetime for the next 20 minutes
ptype=RegExp
pattern=ftpd\[(\d+)\]
context=ftp_$1
continue=TakeNext
desc=event of ftp session $1
action=add ftp_$1 $0; set ftp_$1 1200 (report ftp_$1 /bin/mail root@localhost)
# INACTIVITY_TIMER_host and BAD_LOGIN_FROM_host are created.
# BAD_LOGIN_FROM_host will be used for saving the further BAD LOGIN
# events from that host, and it will have a lifetime of 3 days.
# INACTIVITY_TIMER_host will be used as an inactivity timer, and its
# task is to delete BAD_LOGIN_FROM_host, when that context has not been
# updated with new events for more than 24 hours.
# If the context survives 3 days without having update delays longer
# than 24 hours (i.e., there will be at least one bad login attempt
# from the host per _every_ 24 hour window), then the events saved to
# the context BAD_LOGIN_FROM_host will be reported at the end of the
# third day.
ptype=RegExp
pattern=BAD LOGIN FROM (\S+)
context=!BAD_LOGIN_FROM_$1
continue=TakeNext
desc=First bad login from $1
action=create INACTIVITY_TIMER_$1; create BAD_LOGIN_FROM_$1 259200 \
( report BAD_LOGIN_FROM_$1 /bin/mail root@localhost; \
delete INACTIVITY_TIMER_$1; )
# lifetime of the INACTIVITY_TIMER_host context for another 24 hours
# (note that this will indirectly expand the lifetime of the
# BAD_LOGIN_FROM_host context for the same amount of time)
ptype=RegExp
pattern=BAD LOGIN FROM (\S+)
context=BAD_LOGIN_FROM_$1
desc=Bad login from $1
action=add BAD_LOGIN_FROM_$1 %t: %s; \
set INACTIVITY_TIMER_$1 86400 ( delete BAD_LOGIN_FROM_$1; )
time=0 0 * * *
desc=Sending report
action=shellcmd cat /tmp/myreport | mail root@localhost; \
shellcmd rm -f /tmp/myreport
time=0 0 * * *
desc=Sending report
action=shellcmd cat /tmp/myreport | mail root@localhost && rm -f /tmp/myreport
ptype=substr
pattern=eventA
desc=event A
action=create EVENT_A_APPEARED 300; event CHECK_FOR_CLUSTER
ptype=substr
pattern=eventB
desc=event B
action=create EVENT_B_APPEARED 300; event CHECK_FOR_CLUSTER
ptype=substr
pattern=eventC
desc=event C
action=create EVENT_C_APPEARED 300; event CHECK_FOR_CLUSTER
ptype=substr
pattern=CHECK_FOR_CLUSTER
context=EVENT_A_APPEARED && EVENT_B_APPEARED && EVENT_C_APPEARED
desc=Cluster detected!
action=shellcmd notify.sh %s
ptype=substr
pattern=SEC_STARTUP
context=SEC_INTERNAL_EVENT
desc=Load SNMP module
action=assign %a 0; eval %a (require SNMP); eval %a (exit(1) unless %a)
ptype=substr
pattern=SEC_SHUTDOWN
context=SEC_INTERNAL_EVENT
desc=Write all contexts to stdout
action=eval %o ( while($context = each(%main::context_list)) { \
print "Context name: $context\n"; \
print '-' x 60, "\n"; \
foreach $line (@{$main::context_list{$context}->{"Buffer"}}) { \
print $line, "\n"; \
} \
print '=' x 60, "\n"; \
} )
ptype=perlfunc
pattern=sub { if ($_[0] =~ /(\S+): [Ff]ile system full/) { \
return ($1, $_[1]); } else { return 0; } }
desc=File system $1 full ($2)
action=write -
Section 4. Miscellaneous
Q 4.1: My platform does not have SIGABRT. What signal should I use for
"soft reloads"?