100% CPU on the System caused by Weblogic Process.

Problem: the CPU on the system is pegged to 100% and see application slowdown, and need to trace what is casing this behavior.

Problem Analasys:
In order to properly troubleshoot the high CPU condition Operating System commands are a key to identify the root cause
Please use the following:

1. Use ps and grep to get the PID associated with your userID that started WLS that is using up the CPU.
2. Find the thread id in the WLS process that is using the most CPU via ps -eLo pid,ppid,tid,pcpu,comm | grep .
3. Take several thread dumps of the WebLogic Server process via kill -3 .

NOTE: This plan needs to be executed when the issue is actually happening.


troubleshooting process is explained in detail-->

Unexpected High CPU Usage with WebLogic Server (WLS) Support Pattern (Doc ID 779349.1)

Weblogic 12c AdminServer going to warning state

When you use NodeManage to manage yuor domain, you will notice Admin Server Going to Warning state with STUCK threads.

In the Thread Dump you would see similar to this:

The stuck threads show as 
^-- Holding lock: weblogic.nodemanager.client.SSLClient@1c793422[thin lock]
  weblogic.nodemanager.mbean.StartRequest.start(StartRequest.java:75)
  weblogic.nodemanager.mbean.StartRequest.execute(StartRequest.java:45)

while the AdminServer is waiting on  nodemanager connection, This problem is due to misconfiguration of the Node Manager. If you use startScriptEnabled=true,  need to use -Dweblogic.nodemanager.ServiceEnabled=true in the server start up settings.

Cannot Restart Managed Server : Weblogic because of Exception while getting default policy Provider

While starting the Weblogic Managed server or admin server I was getting the eorror:

Reason: weblogic.security.SecurityInitializationException: The loading of OPSS java security policy provider failed due to exception, see the exception stack trace or the server log file for root cause. If still see no obvious cause, enable the debug flag -Djava.security.debug=jpspolicy to get more information. Error message: oracle.security.jps.JpsException: [PolicyUtil] Exception while getting default policy Provider


weblogic.security.SecurityInitializationException: The loading of OPSS java security policy provider failed due to exception, see the exception stack trace or the server log file for root cause.
 
Caused By: oracle.security.jps.JpsException: [PolicyUtil] Exception while getting default policy Provider
Caused By: java.security.PrivilegedActionException: oracle.security.jps.JpsException: [PolicyUtil] Unable to obtain default JPS Context!
Caused By: oracle.security.jps.JpsException: [PolicyUtil] Unable to obtain default JPS Context!
Caused By: oracle.security.jps.JpsRuntimeException: Cannot read from policy store.
Caused By: oracle.security.jps.JpsRuntimeException: javax.xml.stream.XMLStreamException: javax.xml.stream.XMLStreamException: Premature end of file encountered
 
 
Cause:
 
this can happen when a non weblogic user tried to restart the managed server [may be root by mistake]
 
In my env, it had corrupted the : system-jazn-data.xml [got 0 bytes in it]
 
so i had to restore it :


from : /app/oracle/product/wcpoc/11.1.0/oracle_common/modules/oracle.jps_11.1.1/domain_config/system-jazn-data.xml

to /app/oracle/product/wcpoc/11.1.0/user_projects/domains/wc_poc/config/fmwconfig

and was finally able to restart the managed servers.
 
thanks to Oracle metalink: ID 1348704.1

Auto Deployemnets-using Ant

upload="true"

name="${deploy.name.1}"

source="${deploy.source.1}"

user="${wls.username}"

password="${wls.password}"

verbose="true"

adminurl="t3://${wls.hostname}:${wls.port}" targets="${deploy.target}" />

upload="true"

name="${deploy.name.2}"

source="${deploy.source.2}"

user="${wls.username}"

password="${wls.password}"

verbose="true"

adminurl="t3://${wls.hostname}:${wls.port}" targets="${deploy.target}" />

name="${deploy.name.1}"

failonerror="false"

user="${wls.username}"

password="${wls.password}"

verbose="true"

adminurl="t3://${wls.hostname}:${wls.port}" targets="${deploy.target}" />

name="${deploy.name.2}"

failonerror="false"

user="${wls.username}"

password="${wls.password}"

verbose="true"

adminurl="t3://${wls.hostname}:${wls.port}" targets="${deploy.target}" />

How to add a user to to Administrator in Webcneter

you need to add the user to the role name :

s8bba98ff_4cbb_40b8_beee_296c916a23ed#-#Administrator

How to Configure RSS feeds for Webcenter

If you planning on subcribing to external RSS in webcenter you need to configure the proxy from the server you have configured webcenter.

i) go to the /app/oracle/product/Middleware/Oracle_WC1/common/bin
ii) ./wlst.sh
Welcome to WebLogic Server Administration Scripting Shell
Type help() for help on available commands
wls:/offline>

iii) connect ()

Please enter your username [weblogic] :weblogicPlease enter your password [welcome1] :Please enter your server URL [t3://localhost:7001] :t3://soalab01.ros.com:7001Connecting to t3://soalab01.ros.com:7001 with userid weblogic ...Successfully connected to Admin Server 'AdminServer' that belongs to domain 'wc_domain'.
Warning: An insecure protocol was used to connect to theserver. To ensure on-the-wire security, the SSL port orAdmin port should be used instead.
wls:/wc_domain/serverConfig>

iv) setRssProxyConfig( appName = 'webcenter', proxyHost = 'proxy.ros.com', proxyPort = '80' )

v) restart the Managed Service [ I have restarted both spaces and webcenter services]

Weblogic Server Shutting down automatically - BEA-000388

The WebLogic Server is shutting down automatically after few hrs(7-8) with the following message:


----If you start the $MIDDLEWARE_HOME/user_projects/domains/wc_domain/bin/startWebLogic.sh and put in the background, after few hours of incativity the process shuts down Automaticall.

the cause could be OS specific kicking out the user, so a as to make the startup script independent of this OS specific bootouts , add the folliwing in your startup script startWebLogic.sh

SAVE_JAVA_OPTIONS="-Xrs"

the version i have is:
WebLogic Server Version: 10.3.2.0

Oracle B2B setup over AS2

As2 is a protocol used to transfer EDI payloads over HTTP or HTTPS , the AS2 Headers will add extra level of security while exchanging Data with your Tradin Partners.

Key Advantage of using AS2 is the USe of MDN's - which are Responses for the message sent..similar to ACK that the message is been recieved from a trusted party.

Host TP -------Payload over AS2----------->Remote TP
Host TP <-------MDN for the sent Payload---Remote TP

Follow the steps below to Setup Transfers over AS2 using Oracle B2B Server:

i) Exchange AS2 related information with your Trading Partners [ AS2 Identifiers, End point URL's]
ii)Exchange Public Certificates of Each Parties Involved. [Verisign,godaddy..]

How to Setup Oracle Webcenter Spaces

1. Installation
2. Post Installation
3. Wire up the Webcenter Services with Webcenter Spaces
4. Patches [MLR] Installed :9308777 ,9313626

How to Extend a Table Space

If you run out of Table Space on any of your database
you can extend with the following SQL:

ALTER DATABASE DATAFILE '+SOAUAT_DATA01/idmuat/datafile/idmdata_ts.433.676232907'
AUTOEXTEND ON
NEXT 1000M
MAXSIZE 17000M;
commit;

Truncate Queue Table

#!/bin/bash
export ORAENV_ASK=NO
export JAVA_HOME=/app/oracle/jdk1.5.0_12
#export GETSQLPLUS_HOME=/opt/app/oracle
export ORACLE_SID=SOASTG1
qtabs=`
sqlplus -s / as sysdba<set heading off
set feedback off
set echo off
select distinct queue_table from dba_queue_tables where owner = 'AQUSER';
quit;
EOF
`
for t in $qtabs;do
date
echo "Truncating Queue $t"
sqlplus -s / as sysdba<set heading off
set feedback off
set echo off
declare
po_t dbms_aqadm.aq\$_purge_options_t;
begin
po_t.block := FALSE;
dbms_aqadm.purge_queue_table(queue_table => 'aquser.$t',
purge_condition => '',
purge_options => po_t);
end;
/
COMMIT;
quit;
EOF
done
echo "DONE!"

ORACLE BPEL

i) Out Of Memory Exception in BPEL

Uncaught exception java.lang.OutOfMemoryError in JCA-work-instance:JMSAdapter-47 - cause: Java heap space--

we see this exception when the Resource Adapter was not able to complete the transaction in a defined amount of time., make sure if the transaction timeout is around transaction-timeout="7200" [transaction-timeout is in seconds and defaults to 30]

you can find this parameter : $ORACLE_HOME/j2ee/OC4J_SOA/config/transaction-manager.xml

Useful Unix Commands

#To find and Replace a string recursivly
find . -name \*.txt -type f -exec sed -i 's/Test1/Test2/g' {} \;

#To Uncompress a tar.gz file
gunzip -c filename.tar.gz tar xvf -

# To Search the file excluding the # [Comments ]mark in it
grep -c -v "^#" /opt/irisa/.fetch*rc?


#To Insert a New Line above a PATTERN Match:
sed '/connection-pool>/i\n' test
this will insert two lines on top of matched patter "/connection-pool>"