Progress

Working on Pre-sales

code format="sql" CREATE TABLE DBUSER (  USER_ID       NUMBER ( 5 )    NOT NULL,   USERNAME      VARCHAR2 ( 20 )  NOT NULL ,   CREATED_BY    VARCHAR2 ( 20 )  NOT NULL ,   CREATED_DATE  DATE          NOT NULL ,   PRIMARY KEY ( USER_ID ) ) code code format="sql" CREATE OR REPLACE PROCEDURE getDBUSERByUserId ( 	  p_userid IN DBUSER . USER_ID% TYPE, 	   o_username OUT DBUSER . USERNAME% TYPE , 	   o_createdby OUT  DBUSER . CREATED_BY% TYPE , 	   o_date OUT DBUSER . CREATED_DATE% TYPE ) IS BEGIN

SELECT USERNAME, CREATED_BY , CREATED_DATE INTO o_username, o_createdby , o_date FROM DBUSER WHERE USER_ID = p_userid;

END ; code

code format="sql" DECLARE o_username DBUSER. USERNAME% TYPE ; o_createdby DBUSER. CREATED_BY% TYPE ; o_date DBUSER. CREATED_DATE% TYPE ; BEGIN

getDBUSERByUserId ( 1001, o_username , o_createdby , o_date ) ;

DBMS_OUTPUT. PUT_LINE ( 'username : ' || o_username ) ; DBMS_OUTPUT. PUT_LINE ( 'createdby : ' || o_createdby ) ; DBMS_OUTPUT. PUT_LINE ( 'createddate : ' || o_date ) ;

END ; code

code format="sql" code

[]


 * Teradata Training 28th May 2012**


 * Teradata Architecture:**


 * PE's**
 * AMP's**
 * VDisk**


 * PPI : Partition Primary Index**


 * UPI: Uniq Primary Index**
 * USI: Uniq Secondary Index**


 * Note: don't run the query's untill you see the explain plan on the teradata. On SQL assistant F6 is used to see the stats. F9 to run the sql**


 * NUSI**
 * USI**


 * Residual condition are something that you need to look for.**


 * Note: Good command to get the table structure along with data**
 * create table tablename**
 * as new_table**
 * with data;**

Pig Training - 5th May 2012
 * Hadoop**


 * Analyzing the Job DAG**

Which step/set of jobs consumes most of the time isolate netwroks latencies isolate namenode/jobtracker/Scheduling latencies does the job h as a UDF? understand what each phase, Map and reduce phase does measure the shuffle time use pig provided features if possible EG FR JOIN use illustrator ot eliminate duplicate / unnecessary steps understand and tune the jobs for cluster capacity
 * analyze the jobs counters
 * analyze the key distribution
 * check for skewness
 * estimate the correct number of reducers


 * Specialized JOINS**
 * Combine Small Files
 * processing several small files are inefficient
 * pig.splitCombination=true
 * pig.maxCombinedSplitSize= 
 * care need to be taken if custom loaders are used.
 * Skewed Join
 * parallel joins are vulnerable to the presence of skew in the underlying data
 * if the underlying data is sufficiently skewed, load imbalances will swamp any of the parallelism gains
 * fragment replicate join
 * if you have a small(<100M) table then
 * J = join big by $0, small by $0 u sing replicated
 * Distribute Processing of huge files


 * Data Problems**


 * Pig Bugs**

14th March 2011

Understanding the data model for My organizaiton. Things to remember, try to get the datamodel. Understand the base tables. Understand how the data is being populated across different tables. understand the Primary key and relations ships with other tables

VSP 14th Sept.

Currently working on Implementing the 3 - Tier Architecture for the Developement Region.

 Datastagea Architectur.

The DataStage Enterprise Packs and Change Data Capture components are available in version 8 as shown in the version 8 architecture overview: 

__Today I am Installing Fix Pack for SAP on Data Stage. That when fine...__

__ $ pwd  __ __ /opt/IBM/InformationServer/Server/Projects/test  __ __   I have three DSParams here... $ ls -ltr | grep 'Params' __ __ -rwxrwxr-x 1 dsadm dstage 29374 Nov 23 18:49 DSMFParamsOS390  __ __ -rwxrwxr-x 1 dsadm dstage 44421 Nov 23 18:49 DSMFParams  __ __ -rwxrwxr-x 1 dsadm dstage 53118 Jan 11 15:04 DSParams  __ __ Here the output from the DSParams by grep on 'Logging'$ cat DSParams | grep 'Logging'  __

__ RTLogging=1  __ __ ORLogging=0  __ __ $  __

__ V ersion of UNIX we are using. $ uname -a __ __ AIX bi-etl-deva 1 6 0003D7B9D600 __ _

Running command export TZ=PST8PDT sets TZ in POSIX notation

_ Sreedhar , keep in mind at 8.1 fp1 there is a new stage called the db2 connector stage. This still runs in parallel however the configuration is much easier to setup as compared to the db2 ee stage. It does not require special configuration. Such as the px engine to be copied over available on the remote db2 databases and etc .. Regards Todd Maske _ Please try the following: The Agent process should be listening on port 31531 1) Run this command "netstat -an | grep 31531" to make sure it is listening. 2) Also make sure that the Agent process is running "ps -ef | grep Agent" 3) cd /opt/IBM/InformationServer/ASBNode/bin ./AgentConfig.sh -listAllAgents -user isadmin -password

Hi Sreedhar, We need to get the ASBagent up and running in order for IA to work.

Please try starting the ASBAgent service from the Control Panel ->Services, or from command prompt by running the command 'sc start asbagent'.If that doesn't work, then do the following: Do a 'netstat -an' and check that port 31531 is not being occupied. This is the port that the ASBAgent uses.Then could you do this: 1/ Go into the WebSphere\AppServer\profiles\default\logs\server1 and zip up all the logs. 2/ Delete all the logs WAS logs. 3/ Do a hard reboot (shutdown poweroff) on the AIX box 4/ Once the AIX box has come backup, check if your ASBAgent is running and the port is listening. 5/ If the ASBAgent is not running, then go to a command prompt and run the command 'ps -ef > ps_out.txt'. Then send us this resulting file. 6/ Run the command 'netstat -an > netstat_out.txt' and send us this resulting file as well 7/ Go into the WebSphere\AppServer\profiles\default\logs\server1 and then zip up the newly created logs and send up the old and new logs, please label them accordingly. 8) send the zipped filesIf you want to try to figure out what is happening, have a look in the WAS logs, it will have info on the start up error. You might also try to have a look at the AIX System logs to figure out why the process is not starting/failing.Thank you,

Please make sure that no jobs are running and no users are connected. Perform the following to ensure that none of these processes remain: . 1. ps -ef | grep dsapi 2. ps -ef | grep dscs 3. ps -ef | grep phantom 4. ps -ef | grep dsd 5. ps -ef | grep osh


 * Unlock a Job in DataStage**

If you are not able to see that lock from DS Director, go to DS Administrator , select the project, click on command button, issue command LIST.READU ALL, look for the user and inode for that particular locked process, issue command UNLOCK USER ALL, or UNLOCK ALL.

But, you need DS Admin privilege to do this. The other alternative is to kill that process from unix command line, ps -ef | grep dscs or ps -ef | grep dscs | grep phantom or ps -ef | grep dscs | grep slave get the process id using above command and issue the below command kill -9 Please check whether you are autorized user to use "kill" command or not


 * STOP AND START DATASTAGE ENGINE**

Here are the instructions for restarting the Services, ASBAgent and DataStage engine in IIS v8 (see instructions below for each step). Please be advised that if any other packs are active (SAP R/3 Pack, SAP BW Pack, etc.) you will need to consult the documentation specific to those items for instructions. The order in which to stop all services is:
 * Stop DataStage Engine**
 * Stop the ASB Agent**
 * Stop the WebSphere Application Server (WAS)**

Here are the step-by-step instructions for each of the above. Stop DataStage Engine Log into your UNIX machine as root Make sure that no one is using DataStage. Check if there are any processes left ps -ef | grep -i phantom ps -ef | grep dsapi ps -ef | grep dscs

These commands should not return any results. If they return results, it means that there are still some processes running. You can wait for the users to stop their processes or kill the processes (using kill -9 PID). Check if dsrpc is running

netstat -a | grep dsrpc

If dsrpc is running, then you should get a line with the status LISTEN. If you get more lines it means that there are still some connections and you need to wait. If you see that dsrpc or other connections are in status WAIT then you also need to wait until this it disappears (you can also reboot the machine). Stop DataStage Services using (you have to run this from $DSHOME)

. ./dsenv bin/uv -admin -stop Then see if there is any memory segment left:

ipcs -mop | grep ade

For 64-bit Linux, use: ipcs -qms -a

If you get a result, take note of the PID and remove it (you will need root access)

ipcrm -m PID Then, check the port again using

netstat -a | grep dsrpc

You should not get any result. If you get any results and the dsrpc is in WAIT state, then you need to wait until it disappears (you can also reboot the machine). Stop the ASB Agent (you need to be root user) Go to /IBM/InformationServer/ASBNode/bin Run ./NodeAgents.sh stop

Type 'yes' if you receive the following message: rm: remove write-protected regular file `Agent.pid'? Check the agent has stopped:

ps -ef | grep -i agent Stop the WebSphere Application Server (you need to be root user) Go to /IBM/InformationServer/ASBServer/bin Run ./MetadataServer.sh stop Be sure the java processes go away Now check there are no java processes left behind:

> ps -ef | grep java

remove them if any exists with: > kill PID (NO -9) The order in which to start all services is:
 * Start WebSphere Application Server**
 * Start ASB Agent**
 * Start DataStage Engine**

Here are the step-by-step instructions. Start WebSphere Application Server (you need to be root user) Note: for AIX platform, do the following: unset LDR_CNTRL create a file in ASBServer/bin named MetadataServer_env.sh and add the following line: ulimit -n 10240 Go to /IBM/InformationServer/ASBServer/bin Run ./MetadataServer.sh run Start ASB Agent (you need to be root user) Go to /IBM/InformationServer/ASBNode/bin Run ./NodeAgents.sh start Start DataStage Engine Go to $DSHOME . ./dsenv Run bin/uv -admin -start

>> >> >>