A technical security analysis of the snmp daemon on Exadata
Recently I was asked to analyse the security impact of the snmp daemon on a recent Exadata. This system was running Exadata image version 12.1.2.1.3. This blog article gives you an overview of a lot of the things that surround snmp and security.
First of all what packages are installed doing something with snmp? A list can be obtained the following way:
# rpm -qa | grep snmp net-snmp-utils-5.5-54.0.1.el6_7.1.x86_64 net-snmp-libs-5.5-54.0.1.el6_7.1.x86_64 net-snmp-5.5-54.0.1.el6_7.1.x86_64 sas_snmp-14.02-0103.x86_64
Essentially the usual net-snmp packages and a package called ‘sas_snmp’.
A next important thing is how the firewall is configured. However, the default setting of the firewall on the compute nodes with exadata is the firewall turned off:
# iptables -L -v Chain INPUT (policy ACCEPT 437M packets, 216G bytes) pkts bytes target prot opt in out source destination Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 343M packets, 748G bytes) pkts bytes target prot opt in out source destination
So if there is something running that listens to a network port that can benefit ‘attackers’, there is no firewall to stop them.
Next obvious question is what snmp processes are actually running:
# ps -ef |grep snmp root 7088 1 0 Aug16 ? 00:51:32 /usr/sbin/snmpd -LS0-6d -Lf /dev/null -p /var/run/snmpd.pid root 33443 1 0 03:14 ? 00:00:49 /usr/sbin/lsi_mrdsnmpagent -c /etc/snmp/snmpd.conf root 33454 33443 0 03:14 ? 00:00:00 /usr/sbin/lsi_mrdsnmpagent -c /etc/snmp/snmpd.conf
The snmpd process is the net-snmp snmp daemon. However, there are two additional processes running with the name ‘snmp’ in them, one is owned by init, and a processes that this process has spawned. The name ‘lsi_mrdsnmpagent’ probably means LSI MegaRaid SNMP agent. That gives a fair hint this processes is doing something snmp related specifically for the LSI MegaRaid adapter, which is the disk controller.
Are there any open ports related to snmp processes?
# netstat -anp | grep snmp tcp 0 0 127.0.0.1:199 0.0.0.0:* LISTEN 7088/snmpd udp 0 0 0.0.0.0:161 0.0.0.0:* 7088/snmpd udp 0 0 0.0.0.0:22917 0.0.0.0:* 7088/snmpd
1. tcp port 199
This is support for the SMUX protocol (RFC 1227) to communicate with SMUX-based subagents. This is a deprecated feature in favour of AgentX. It is considered a bug (https://bugzilla.redhat.com/show_bug.cgi?id=110931) the daemon is still using this port. However the port is opened on localhost (127.0.0.1) and as such not reachable from outside of the machine, which means it is not a direct security problem.
2. udp port 161
This is the default snmpd port. This port is open to the outside world on the compute node, which can be seen from the address 0.0.0.0 in the above ‘source’ column. The port being open can be verified using another machine and the ‘nmap’ tool:
$ sudo nmap -Pn -sU -p 161 311.1.1.1 Password: Starting Nmap 6.47 ( http://nmap.org ) at 2016-10-26 15:00 CEST Nmap scan report for 311.1.1.1 Host is up (0.087s latency). PORT STATE SERVICE 161/udp open snmp
The status ‘open’ shows this udp port does respond to requests.
3. udp port 22917 (in this case; this port number is random)
This is a random port that gets set for trapsink directive set in the /etc/snmp/snmpd.conf. A trap sink is the destination for snmp traps that get triggered. Although this udp port is in use, it does not respond to network traffic:
$ sudo nmap -Pn -sU -p 22917 311.1.1.1 Password: Starting Nmap 6.47 ( http://nmap.org ) at 2016-10-26 15:22 CEST Nmap scan report for 311.1.1.1 Host is up. PORT STATE SERVICE 22917/udp open|filtered unknown
The status ‘open|filtered’ does mean the udp port does not respond to requests.
Now let’s look how the actual configuration file of the snmp daemon looks like on exadata. The configuration file is /etc/snmp/snmpd.conf:
snmp daemon configuration file: trapcommunity public trapsink 127.0.0.1 public rocommunity public 127.0.0.1 rwcommunity public 127.0.0.1 access RWGroup "" any noauth exact all all all com2sec snmpclient 127.0.0.1 public group RWGroup v1 snmpclient pass .1.3.6.1.4.1.4413.4.1 /usr/bin/ucd5820stat pass .1.3.6.1.4.1.3582 /usr/sbin/lsi_mrdsnmpmain syscontact Root <root@localhost> (configure /etc/snmp/snmp.local.conf) syslocation Unknown (edit /etc/snmp/snmpd.conf) view all included .1 80
The snmpd.conf file shows:
– trapsink destination (127.0.0.1, localhost) and community string (public).
– the ro and rw communities are set to ‘public 127.0.0.1’
In general it is advised to change the community strings to something unique to avoid being easy guessable. However, in this case there’s also a network description following the ro and rw community, which is: 127.0.0.1. This means snmp access is restricted to localhost.
This can be verified by running snmpwalk from another machine:
$ snmpwalk -v 2c -c public 311.1.1.1 Timeout: No Response from 311.1.1.1
This means there is no way to communicate to the snmp daemon from outside of the machine. We can see from the snmp daemon configuration file that access to the snmp deamon is limited to localhost.
It seems the LSI megaraid snmp agent works together with snmpd:
root 33443 1 0 03:14 ? 00:01:01 /usr/sbin/lsi_mrdsnmpagent -c /etc/snmp/snmpd.conf root 33454 33443 0 03:14 ? 00:00:00 /usr/sbin/lsi_mrdsnmpagent -c /etc/snmp/snmpd.conf
Obviously it reads the snmpd.conf (-c /etc/snmp/snmpd.conf as seen above), but it has got a configuration file of its own. This configuration file of ‘lsi_mrdsnmpagent’ process can be seen when looking at the files inside the sas_snmp rpm package (rpm -ql sas_snmp), but the main evidence it is using the file can be derived from looking at the open file descriptors of the lsi_mrdsnmpagent process:
# ls -ls /proc/$(pgrep -f lsi_mrdsnmpagent | head -1)/fd total 0 0 lr-x------. 1 root root 64 Oct 26 03:14 0 -> /dev/null 0 lr-x------. 1 root root 64 Oct 26 03:14 1 -> /etc/lsi_mrdsnmp/sas/sas_TrapDestination.conf 0 l-wx------. 1 root root 64 Oct 26 03:14 2 -> /var/log/cellos/cron_daily_cellos.stderr (deleted) 0 lrwx------. 1 root root 64 Oct 26 03:14 3 -> socket:[2923149143] 0 l-wx------. 1 root root 64 Oct 26 03:14 4 -> /var/log/cellos/cellos.log (deleted) 0 l-wx------. 1 root root 64 Oct 26 03:14 5 -> /var/log/cellos/cellos.trc (deleted) 0 lr-x------. 1 root root 64 Oct 26 03:14 6 -> /etc/snmp/snmpd.conf 0 lr-x------. 1 root root 64 Oct 26 03:14 7 -> /etc/redhat-release 0 lr-x------. 1 root root 64 Oct 26 03:14 8 -> /dev/megaraid_sas_ioctl_node 0 lr-x------. 1 root root 64 Oct 26 03:14 9 -> pipe:[2919419375]
Line 4 shows ‘/etc/lsi_mrdsnmp/sas/sas_TrapDestination.conf’! Let’s look inside that configuration file:
# cat /etc/lsi_mrdsnmp/sas/sas_TrapDestination.conf ################################################# # Agent Service needs the IP addresses to sent trap # The trap destination may be specified in this file or # using snmpd.conf file. Following indicators can be set # on "TrapDestInd" to instruct the agent to pick the IPs # as the destination. # 1 - IPs only from snmpd.conf # 2 - IPs from this file only # 3 - IPs from both the files ################################################# TrapDestInd 3 #############Trap Destination IP################## # Add port no after IP address with no space after # colon to send the SNMP trap message to custom port. # Community is to be mentioned after IP. If no community # is mentioned, default SNMP community 'public' shall be # used. 'trapcommunity' token is also used in snmpd.conf. # Alternatively, you can also use trapsink command # in snmpd.conf to send the SNMP trap message to # custom port, else default SNMP trap port '162' shall # be used. 127.0.0.1 public # 145.147.201.88:1234 public # 145.146.180.20:3061 testComm 127.0.0.1:8162 public
It is a configuration file that works alongside the snmpd.conf configuration. What is important to see, is ‘TrapDestInd’, which is set at ‘3’, which means that traps are send to trap destinations set in the snmpd.conf file AND set in the sas_TrapDestionation.conf file. Two traps are defined in the file, 127.0.0.1 with community string public, which means it sends a trap to udp port 161 (at which the snmpd process is listening, as we saw earlier in the open ports list), but the most interesting thing here is there’s also a trap send to 127.0.0.1 at port 8162. That is a port number I do not know from the top of my head!
However, it’s simple to find out. The first thing to check is to see what process is running at port 8162:
# netstat -anp | grep 8162 udp 0 0 :::8162 :::* 15233/java
That’s a java process! Let’s grep the process number to see if the full command line gives more clues what this java process is:
# ps -ef | grep 15233 dbmsvc 15233 15136 0 Aug16 ? 05:32:25 /usr/java/jdk1.7.0_80/bin/java -client -Xms256m -Xmx512m -XX:CompileThreshold=8000 -XX:PermSize=128m -XX:MaxPermSize=256m -Dweblogic.Name=msServer -Djava.security.policy=/opt/oracle/dbserver_12.1.2.1.3.151021/dbms/deploy/wls/wlserver_10.3/server/lib/weblogic.policy -XX:-UseLargePages -XX:ParallelGCThreads=8 -Dweblogic.ListenPort=7878 -Djava.security.egd=file:/dev/./urandom -Xverify:none -da -Dplatform.home=/opt/oracle/dbserver_12.1.2.1.3.151021/dbms/deploy/wls/wlserver_10.3 -Dwls.home=/opt/oracle/dbserver_12.1.2.1.3.151021/dbms/deploy/wls/wlserver_10.3/server -Dweblogic.home=/opt/oracle/dbserver_12.1.2.1.3.151021/dbms/deploy/wls/wlserver_10.3/server -Dweblogic.management.discover=true -Dwlw.iterativeDev= -Dwlw.testConsole= -Dwlw.logErrorsToConsole= -Dweblogic.ext.dirs=/opt/oracle/dbserver_12.1.2.1.3.151021/dbms/deploy/wls/patch_wls1036/profiles/default/sysext_manifest_classpath weblogic.Server
That’s java running weblogic, with the name ‘msServer’. That is something that is part of the daemons that serve dbmcli (alike the daemons that service cellcli on the cells)!
This actually makes sense. The daemons that manage the database server fetch hardware status information and hardware failures from the BMC using the IPMI device (/dev/ipmi0). However the LSI MegaRaid adapter can not provide its status in that way. So in order for the management daemons to keep track of events on the LSI MegaRaid adapter (hardware issues), a daemon that works together with the snmp daemon is setup, which sends snmp traps if something occurs. The management daemon has setup a port that listens for these traps.
I do not know if the community strings is processed by the management deamon processing the trap. However, the port number on which the daemon is listening for traps is defined in ‘/opt/oracle/dbserver/dbms/deploy/config/cellinit.ora’ with the directive BMC_SNMP_PORT.
Conclusion
As far as I can see, the reason the snmp deamon is running is to be able to run the LSI MegaRaid SNMP agent process, so it can send traps to the compute node’s management daemons. Since most Exadata compute nodes do not have the firewall enabled, udp port 161 is exposed. The settings of the snmp daemon itself limits access to localhost.
Frits Hoogland Weblog 
Pingback: Exadata and WebLogic CVE-2019-2725 – Andy Colvin's Oracle Blog