Skip to content

Commit 15ed2ce

Browse files
cwadhwani-splunkrenovate[bot]wojtekzylamstopa-splunksbylica-splunk
authored
feat: Release 3.33 (#2654)
* fix: Non pri cef fix (#2641) * chore(deps): update dependency mkdocs-material to v9.5.42 (#2624) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * chore(deps): update splunk/addonfactory-test-matrix-action action to v2.1.9 (#2620) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * fix: fix CISE_Alarm messages parsing (#2609) * fix: improve SC4S Dashboard performance (#2592) * docs: Removed reference of Cisco eStreamer for Splunk app from ASA/FTD doc (#2629) * docs: Removed reference of Cisco eStreamer for Splunk app * fix: Updated the regex for non pri cef formated logs such that a '<space>digit' date will be accepted * Updated the regex with some more constraints --------- Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: wojtekzyla <[email protected]> Co-authored-by: mstopa-splunk <[email protected]> * chore(deps): update dependency mkdocs-material to v9.5.47 (#2644) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * feat: Netapp ontap audit ems support (#2639) * chore(deps): update dependency mkdocs-material to v9.5.42 (#2624) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * chore(deps): update splunk/addonfactory-test-matrix-action action to v2.1.9 (#2620) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * fix: fix CISE_Alarm messages parsing (#2609) * fix: improve SC4S Dashboard performance (#2592) * docs: Removed reference of Cisco eStreamer for Splunk app from ASA/FTD doc (#2629) * docs: Removed reference of Cisco eStreamer for Splunk app * feat: Added support for ems logs and fixed the existing classification * Updated the documentation and made some changes in the parser * Updating the test file such that all the test cases are passing * Added support in sc4s lite * docs: Added the migration precaution in the upgrade.md file * Removed an addionally created test file and merged my changes to the existing one * Updated the code to maintain the backward compatibility * Updated the test-container workflow to set the newly introduced environment variable's value to 'yes' and made some changes in the test vps parser. --------- Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: wojtekzyla <[email protected]> Co-authored-by: mstopa-splunk <[email protected]> * Added juniper log parsing * Migrated to restricted sqlite --------- Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: wojtekzyla <[email protected]> Co-authored-by: mstopa-splunk <[email protected]> Co-authored-by: sbylica-splunk <[email protected]> Co-authored-by: Szymon Bylica <[email protected]>
1 parent 6da3060 commit 15ed2ce

23 files changed

+348
-104
lines changed

‎.github/workflows/ci-lite.yaml

+1
Original file line numberDiff line numberDiff line change
@@ -227,6 +227,7 @@ jobs:
227227
TEST_SC4S_ACTIVATE_EXAMPLES: "yes"
228228
SC4S_DEBUG_CONTAINER: "yes"
229229
SC4S_SOURCE_VMWARE_VSPHERE_GROUPMSG: "yes"
230+
SC4S_NETAPP_ONTAP_NEW_FORMAT: "yes"
230231
SC4S_USE_VPS_CACHE: "yes"
231232
steps:
232233
- name: Checkout

‎.github/workflows/ci-main.yaml

+1
Original file line numberDiff line numberDiff line change
@@ -227,6 +227,7 @@ jobs:
227227
TEST_SC4S_ACTIVATE_EXAMPLES: "yes"
228228
SC4S_DEBUG_CONTAINER: "yes"
229229
SC4S_SOURCE_VMWARE_VSPHERE_GROUPMSG: "yes"
230+
SC4S_NETAPP_ONTAP_NEW_FORMAT: "yes"
230231
SC4S_USE_VPS_CACHE: "yes"
231232

232233
steps:

‎docs/sources/vendor/NetApp/ontap.md

+41-3
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
## Key facts
44

55
* MSG Format based filter
6-
* Legacy BSD Format default port 514
6+
* Netapp Ontap messages are not distinctive. So, either configure known Netapp Ontap hosts in SC4S, or open unique ports for Netapp Ontap devices
77

88
## Links
99

@@ -16,11 +16,49 @@
1616

1717
| sourcetype | notes |
1818
|----------------|---------------------------------------------------------------------------------------------------------|
19-
| netapp:ems | None |
19+
| ontap:ems | This sourcetype will be assinged only when the environment variable `SC4S_NETAPP_ONTAP_NEW_FORMAT` is not set or is set to 'no'. By default it is unset |
20+
| netapp:ontap:audit | This sourcetype will be assinged only when the environment variable `SC4S_NETAPP_ONTAP_NEW_FORMAT` is set to 'yes' |
21+
| netapp:ontap:ems | This sourcetype will be assinged only when the environment variable `SC4S_NETAPP_ONTAP_NEW_FORMAT` is set to 'yes' |
2022

2123
## Sourcetype and Index Configuration
2224

2325
| key | sourcetype | index | notes |
2426
|----------------|----------------|----------------|----------------|
25-
| netapp_ontap | netapp:ems | infraops | none |
27+
| netapp_ontap | ontap:ems | infraops | none |
28+
| netapp_ontap_audit | netapp:ontap:audit | infraops | none |
29+
| netapp_ontap_ems | netapp:ontap:ems | infraops | none |
2630

31+
## Options
32+
33+
| Variable | default | description |
34+
|----------------|----------------|----------------|
35+
| SC4S_NETAPP_ONTAP_NEW_FORMAT | empty string | (empty/yes) Set to "yes" for the applying the latest changes. Make sure to configure your system to send the logs to a specific port or have a hostname-based configuration |
36+
37+
## Parser Configuration
38+
1. Through sc4s-vps
39+
```c
40+
#/opt/sc4s/local/config/app-parsers/app-vps-netapp_ontap.conf
41+
#File name provided is a suggestion it must be globally unique
42+
43+
application app-vps-test-netapp_ontap[sc4s-vps] {
44+
filter {
45+
host("netapp-ontap-" type(string) flags(prefix))
46+
or (
47+
message("netapp-ontap-" type(string) flags(prefix))
48+
and program("netapp-ontap-" type(string) flags(prefix))
49+
)
50+
};
51+
parser {
52+
p_set_netsource_fields(
53+
vendor('netapp')
54+
product('ontap')
55+
);
56+
};
57+
};
58+
```
59+
60+
2. or through unique port
61+
```
62+
# /opt/sc4s/env_file
63+
SC4S_LISTEN_NETAPP_ONTAP_UDP_PORT=5005
64+
```

‎docs/upgrade.md

+3
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,9 @@ For a step by step guide [see here](./v3_upgrade.md).
1818

1919
You may need to migrate legacy log paths or version 1 app-parsers for version 2. To do this, open an issue and attach the original configuration and a compressed pcap of sample data for testing. We will evaluate whether to include the source in an upcoming release.
2020

21+
### Upgrade from <3.33.0
22+
In NetApp ONTAP, the ontap:ems sourcetype has been updated to netapp:ontap:audit, so old logs are now classified under netapp:ontap:audit. Additionally, a new netapp:ontap:ems sourcetype has been introduced. If you upgrade and want these new changes, ensure that you set `SC4S_NETAPP_ONTAP_NEW_FORMAT` environment variable to `yes` and configure your system to send the logs to a specific port or have a hostname-based configuration in place for proper log onboarding into Splunk.
23+
2124
### Upgrade from <2.23.0
2225

2326
* In VMware vSphere, update the ESX and vCenter sourcetype for add-on compatibility.

‎package/etc/conf.d/conflib/netsource/app-netsource-netapp_ontap.conf

+55
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
block parser app-netsource-netapp_ontap() {
2+
channel {
3+
rewrite {
4+
r_set_splunk_dest_default(
5+
index("infraops")
6+
vendor("netapp")
7+
product("ontap")
8+
);
9+
};
10+
11+
if {
12+
parser {
13+
regexp-parser(
14+
prefix(".tmp.")
15+
patterns('^[A-Za-z0-9\-\_\.]+: [0-9a-f]+\.[0-9a-f]+ [0-9a-f]+ [A-Z][a-z][a-z] (?<timestamp>[A-Z][a-z][a-z] \d\d \d\d\d\d \d\d:\d\d:\d\d [+-]?\d{1,2}:\d\d)')
16+
);
17+
date-parser-nofilter(
18+
format(
19+
'%b %d %Y %H:%M:%S %z',
20+
)
21+
template("${.tmp.timestamp}")
22+
);
23+
};
24+
25+
rewrite {
26+
set('$PROGRAM: $MESSAGE', value(MESSAGE));
27+
set('$PROGRAM', value(HOST));
28+
unset(value(PROGRAM));
29+
};
30+
31+
rewrite {
32+
r_set_splunk_dest_update_v2(
33+
sourcetype('netapp:ontap:audit')
34+
class('audit')
35+
);
36+
};
37+
} else {
38+
rewrite {
39+
r_set_splunk_dest_update_v2(
40+
sourcetype('netapp:ontap:ems')
41+
class('ems')
42+
);
43+
};
44+
};
45+
};
46+
};
47+
48+
application app-netsource-netapp_ontap[sc4s-network-source] {
49+
filter {
50+
match("netapp", value('.netsource.sc4s_vendor'), type(string))
51+
and match("ontap", value('.netsource.sc4s_product'), type(string))
52+
and "`SC4S_NETAPP_ONTAP_NEW_FORMAT`" eq "yes"
53+
};
54+
parser { app-netsource-netapp_ontap(); };
55+
};

‎package/etc/conf.d/conflib/raw/app-raw-bsd_nopri.conf

+1-1
Original file line numberDiff line numberDiff line change
@@ -53,7 +53,7 @@ block parser app-raw-bsd_nopri() {
5353
};
5454
application app-raw-bsd_nopri[sc4s-raw-syslog] {
5555
filter {
56-
message('^\w\w\w \d\d \d\d:\d\d:\d\d ');
56+
message('^\w{3} ([0 ][1-9]|[12]\d|3[01]) (0\d|1\d|2[0-3]):([0-5]\d):([0-5]\d) ');
5757
};
5858
parser { app-raw-bsd_nopri(); };
5959
};

‎package/etc/conf.d/conflib/syslog/app-syslog-juniper_junos_unstructured.conf

+1
Original file line numberDiff line numberDiff line change
@@ -74,6 +74,7 @@ application app-syslog-juniper_junos_unstructured-pgm[sc4s-syslog-pgm] {
7474
or program('RT_FLOW' type(string) flags(prefix))
7575
or program('RT_IDS' type(string) flags(prefix))
7676
or program('RT_UTM' type(string) flags(prefix))
77+
or program('RT_SYSTEM' type(string) flags(prefix))
7778
or program('Juniper' type(string) flags(prefix))
7879
or program('rpd' type(string) flags(prefix))
7980
or program('eswd' type(string) flags(prefix))

‎package/etc/conf.d/conflib/syslog/app-syslog-netapp_ontap.conf

+2-1
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,8 @@ block parser app-syslog-netapp_ontap() {
3737
};
3838
application app-syslog-netapp_ontap[sc4s-syslog] {
3939
filter {
40-
program('^[A-Za-z0-9\-\_\.]+$');
40+
program('^[A-Za-z0-9\-\_\.]+$')
41+
and not "`SC4S_NETAPP_ONTAP_NEW_FORMAT`" eq "yes";
4142
};
4243
parser { app-syslog-netapp_ontap(); };
4344
};

‎package/etc/pylib/parser_source_cache.py

+9-13
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22
import traceback
33
import socket
44
import struct
5+
from sqlitedict import SqliteDict
56

67
import time
78

@@ -16,14 +17,15 @@ class LogParser:
1617
class LogDestination:
1718
pass
1819

20+
1921
def ip2int(addr):
2022
ip4_to_int = lambda addr: struct.unpack("!I", socket.inet_aton(addr))[0]
21-
23+
2224
def ip6_to_int(addr):
2325
ip6 = socket.inet_pton(socket.AF_INET6, addr)
2426
a, b = struct.unpack(">QQ", ip6)
2527
return (a << 64) | b
26-
28+
2729
try:
2830
return ip4_to_int(addr)
2931
except OSError:
@@ -39,7 +41,7 @@ def int_to_ip6(num):
3941
ip6 = struct.pack(">QQ", a, b)
4042
addr = socket.inet_ntop(socket.AF_INET6, ip6)
4143
return addr
42-
44+
4345
try:
4446
return int_to_ip4(addr)
4547
except struct.error:
@@ -51,10 +53,8 @@ def int_to_ip6(num):
5153

5254
class psc_parse(LogParser):
5355
def init(self, options):
54-
from sqlite_utils import RestrictedSqliteDict
55-
5656
self.logger = syslogng.Logger()
57-
self.db = RestrictedSqliteDict(f"{hostdict}.sqlite")
57+
self.db = SqliteDict(f"{hostdict}.sqlite")
5858
return True
5959

6060
def deinit(self):
@@ -80,11 +80,9 @@ def parse(self, log_message):
8080

8181
class psc_dest(LogDestination):
8282
def init(self, options):
83-
from sqlite_utils import RestrictedSqliteDict
84-
8583
self.logger = syslogng.Logger()
8684
try:
87-
self.db = RestrictedSqliteDict(f"{hostdict}.sqlite", autocommit=True)
85+
self.db = SqliteDict(f"{hostdict}.sqlite", autocommit=True)
8886
except Exception:
8987
exc_type, exc_value, exc_traceback = sys.exc_info()
9088
lines = traceback.format_exception(exc_type, exc_value, exc_traceback)
@@ -125,9 +123,7 @@ def flush(self):
125123

126124

127125
if __name__ == "__main__":
128-
from sqlite_utils import RestrictedSqliteDict
129-
130-
db = RestrictedSqliteDict(f"{hostdict}.sqlite", autocommit=True)
126+
db = SqliteDict(f"{hostdict}.sqlite", autocommit=True)
131127
db[0] = "seed"
132128
db.commit()
133-
db.close()
129+
db.close()

‎package/etc/pylib/parser_vps_cache.py

+5-7
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22
import traceback
33
import socket
44
import struct
5+
from sqlitedict import SqliteDict
56

67
import time
78

@@ -16,15 +17,14 @@ class LogParser:
1617
class LogDestination:
1718
pass
1819

20+
1921
hostdict = str("/var/lib/syslog-ng/vps")
2022

2123

2224
class vpsc_parse(LogParser):
2325
def init(self, options):
24-
from sqlite_utils import RestrictedSqliteDict
25-
2626
self.logger = syslogng.Logger()
27-
self.db = RestrictedSqliteDict(f"{hostdict}.sqlite")
27+
self.db = SqliteDict(f"{hostdict}.sqlite")
2828
return True
2929

3030
def deinit(self):
@@ -50,11 +50,9 @@ def parse(self, log_message):
5050

5151
class vpsc_dest(LogDestination):
5252
def init(self, options):
53-
from sqlite_utils import RestrictedSqliteDict
54-
5553
self.logger = syslogng.Logger()
5654
try:
57-
self.db = RestrictedSqliteDict(f"{hostdict}.sqlite", autocommit=True)
55+
self.db = SqliteDict(f"{hostdict}.sqlite", autocommit=True)
5856
except Exception:
5957
exc_type, exc_value, exc_traceback = sys.exc_info()
6058
lines = traceback.format_exception(exc_type, exc_value, exc_traceback)
@@ -100,4 +98,4 @@ def flush(self):
10098

10199

102100
if __name__ == "__main__":
103-
pass
101+
pass

‎package/etc/pylib/psc_dump.py

+2-3
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,3 @@
1-
21
import sys
32
import traceback
43
import socket
@@ -9,5 +8,5 @@
98
hostdict = str("/var/lib/syslog-ng/cache/hostip")
109
db = SqliteDict(f"{hostdict}.sqlite")
1110

12-
for k,v in db.items():
13-
print(f"key={k}={v}")
11+
for k, v in db.items():
12+
print(f"key={k}={v}")

‎package/etc/pylib/sqlite_utils.py

-28
This file was deleted.

‎package/etc/test_parsers/app-vps-test-netapp_ontap.conf

+15
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
application app-vps-test-netapp_ontap[sc4s-vps] {
2+
filter {
3+
host("netapp-ontap-" type(string) flags(prefix))
4+
or (
5+
message("netapp-ontap-" type(string) flags(prefix))
6+
and program("netapp-ontap-" type(string) flags(prefix))
7+
)
8+
};
9+
parser {
10+
p_set_netsource_fields(
11+
vendor('netapp')
12+
product('ontap')
13+
);
14+
};
15+
};

‎package/lite/etc/addons/juniper/app-syslog-juniper_junos_unstructured.conf

+1
Original file line numberDiff line numberDiff line change
@@ -74,6 +74,7 @@ application app-syslog-juniper_junos_unstructured-pgm[sc4s-syslog-pgm] {
7474
or program('RT_FLOW' type(string) flags(prefix))
7575
or program('RT_IDS' type(string) flags(prefix))
7676
or program('RT_UTM' type(string) flags(prefix))
77+
or program('RT_SYSTEM' type(string) flags(prefix))
7778
or program('Juniper' type(string) flags(prefix))
7879
or program('rpd' type(string) flags(prefix))
7980
or program('eswd' type(string) flags(prefix))

0 commit comments

Comments
 (0)