100% CPU on the System caused by Weblogic Process.

Problem: the CPU on the system is pegged to 100% and see application slowdown, and need to trace what is casing this behavior.

Problem Analasys:
In order to properly troubleshoot the high CPU condition Operating System commands are a key to identify the root cause
Please use the following:

1. Use ps and grep to get the PID associated with your userID that started WLS that is using up the CPU.
2. Find the thread id in the WLS process that is using the most CPU via ps -eLo pid,ppid,tid,pcpu,comm | grep .
3. Take several thread dumps of the WebLogic Server process via kill -3 .

NOTE: This plan needs to be executed when the issue is actually happening.


troubleshooting process is explained in detail-->

Unexpected High CPU Usage with WebLogic Server (WLS) Support Pattern (Doc ID 779349.1)

Weblogic 12c AdminServer going to warning state

When you use NodeManage to manage yuor domain, you will notice Admin Server Going to Warning state with STUCK threads.

In the Thread Dump you would see similar to this:

The stuck threads show as 
^-- Holding lock: weblogic.nodemanager.client.SSLClient@1c793422[thin lock]
  weblogic.nodemanager.mbean.StartRequest.start(StartRequest.java:75)
  weblogic.nodemanager.mbean.StartRequest.execute(StartRequest.java:45)

while the AdminServer is waiting on  nodemanager connection, This problem is due to misconfiguration of the Node Manager. If you use startScriptEnabled=true,  need to use -Dweblogic.nodemanager.ServiceEnabled=true in the server start up settings.

Cannot Restart Managed Server : Weblogic because of Exception while getting default policy Provider

While starting the Weblogic Managed server or admin server I was getting the eorror:

Reason: weblogic.security.SecurityInitializationException: The loading of OPSS java security policy provider failed due to exception, see the exception stack trace or the server log file for root cause. If still see no obvious cause, enable the debug flag -Djava.security.debug=jpspolicy to get more information. Error message: oracle.security.jps.JpsException: [PolicyUtil] Exception while getting default policy Provider


weblogic.security.SecurityInitializationException: The loading of OPSS java security policy provider failed due to exception, see the exception stack trace or the server log file for root cause.
 
Caused By: oracle.security.jps.JpsException: [PolicyUtil] Exception while getting default policy Provider
Caused By: java.security.PrivilegedActionException: oracle.security.jps.JpsException: [PolicyUtil] Unable to obtain default JPS Context!
Caused By: oracle.security.jps.JpsException: [PolicyUtil] Unable to obtain default JPS Context!
Caused By: oracle.security.jps.JpsRuntimeException: Cannot read from policy store.
Caused By: oracle.security.jps.JpsRuntimeException: javax.xml.stream.XMLStreamException: javax.xml.stream.XMLStreamException: Premature end of file encountered
 
 
Cause:
 
this can happen when a non weblogic user tried to restart the managed server [may be root by mistake]
 
In my env, it had corrupted the : system-jazn-data.xml [got 0 bytes in it]
 
so i had to restore it :


from : /app/oracle/product/wcpoc/11.1.0/oracle_common/modules/oracle.jps_11.1.1/domain_config/system-jazn-data.xml

to /app/oracle/product/wcpoc/11.1.0/user_projects/domains/wc_poc/config/fmwconfig

and was finally able to restart the managed servers.
 
thanks to Oracle metalink: ID 1348704.1

Auto Deployemnets-using Ant

upload="true"

name="${deploy.name.1}"

source="${deploy.source.1}"

user="${wls.username}"

password="${wls.password}"

verbose="true"

adminurl="t3://${wls.hostname}:${wls.port}" targets="${deploy.target}" />

upload="true"

name="${deploy.name.2}"

source="${deploy.source.2}"

user="${wls.username}"

password="${wls.password}"

verbose="true"

adminurl="t3://${wls.hostname}:${wls.port}" targets="${deploy.target}" />

name="${deploy.name.1}"

failonerror="false"

user="${wls.username}"

password="${wls.password}"

verbose="true"

adminurl="t3://${wls.hostname}:${wls.port}" targets="${deploy.target}" />

name="${deploy.name.2}"

failonerror="false"

user="${wls.username}"

password="${wls.password}"

verbose="true"

adminurl="t3://${wls.hostname}:${wls.port}" targets="${deploy.target}" />

How to add a user to to Administrator in Webcneter

you need to add the user to the role name :

s8bba98ff_4cbb_40b8_beee_296c916a23ed#-#Administrator

How to Configure RSS feeds for Webcenter

If you planning on subcribing to external RSS in webcenter you need to configure the proxy from the server you have configured webcenter.

i) go to the /app/oracle/product/Middleware/Oracle_WC1/common/bin
ii) ./wlst.sh
Welcome to WebLogic Server Administration Scripting Shell
Type help() for help on available commands
wls:/offline>

iii) connect ()

Please enter your username [weblogic] :weblogicPlease enter your password [welcome1] :Please enter your server URL [t3://localhost:7001] :t3://soalab01.ros.com:7001Connecting to t3://soalab01.ros.com:7001 with userid weblogic ...Successfully connected to Admin Server 'AdminServer' that belongs to domain 'wc_domain'.
Warning: An insecure protocol was used to connect to theserver. To ensure on-the-wire security, the SSL port orAdmin port should be used instead.
wls:/wc_domain/serverConfig>

iv) setRssProxyConfig( appName = 'webcenter', proxyHost = 'proxy.ros.com', proxyPort = '80' )

v) restart the Managed Service [ I have restarted both spaces and webcenter services]

Weblogic Server Shutting down automatically - BEA-000388

The WebLogic Server is shutting down automatically after few hrs(7-8) with the following message:


----If you start the $MIDDLEWARE_HOME/user_projects/domains/wc_domain/bin/startWebLogic.sh and put in the background, after few hours of incativity the process shuts down Automaticall.

the cause could be OS specific kicking out the user, so a as to make the startup script independent of this OS specific bootouts , add the folliwing in your startup script startWebLogic.sh

SAVE_JAVA_OPTIONS="-Xrs"

the version i have is:
WebLogic Server Version: 10.3.2.0