Tuesday, August 13, 2019

I am Going to Start DataStage Training online sessions will start soon @Ganeswarreddy,Ph:+971-586938453/Whats app :+91-9167232488

I am Going to Start DataStage Training online sessions will start soon @Ganeswarreddy,Ph:+971-586938453/Whats app :+91-9167232488



Interested Candidates Send me request to below mail id:

ganeswarreddy.etl@gmail.com

Monday, August 12, 2019

List of Environment Variables in DataStage

List of Environment Variables in DataStage


General Job Administration

APT_CHECKPOINT_DIR
APT_CLOBBER_OUTPUT
APT_CONFIG_FILE
APT_DISABLE_COMBINATION
APT_EXECUTION_MODE
APT_ORCHHOME
APT_STARTUP_SCRIPT
APT_NO_STARTUP_SCRIPT
APT_STARTUP_STATUS
APT_THIN_SCORE

Job Monitoring

APT_MONITOR_SIZE
APT_MONITOR_TIME
APT_NO_JOBMON
APT_PERFORMANCE_DATA

Buffering

APT_BUFFER_FREE_RUN
APT_BUFFER_MAXIMUM_MEMORY
APT_BUFFER_MAXIMUM_TIMEOUT
APT_BUFFER_DISK_WRITE_INCREMENT
APT_BUFFERING_POLICY
APT_DISABLE_ROOT_FORKJOIN
APT_SHARED_MEMORY_BUFFERS

Compiler

APT_COMPILER
APT_COMPILEOPT
APT_LINKER
APT_LINKOPT

Debugging

APT_DEBUG_OPERATOR
APT_DEBUG_MODULE_NAMES
APT_DEBUG_PARTITION
APT_DEBUG_SIGNALS
APT_DEBUG_STEP
APT_DEBUG_SUBPROC
APT_EXECUTION_MODE
APT_PM_DBX
APT_PM_GDB
APT_PM_SHOW_PIDS
APT_PM_XLDB
APT_PM_XTERM
APT_PXDEBUGGER_FORCE_SEQUENTIAL
APT_SHOW_LIBLOAD

Partitioning

APT_NO_PART_INSERTION
APT_NO_PARTSORT_OPTIMIZATION
APT_PARTITION_COUNT
APT_PARTITION_NUMBER

Reading and Writing Files

APT_DELIMITED_READ_SIZE
APT_FILE_IMPORT_BUFFER_SIZE
APT_FILE_EXPORT_BUFFER_SIZE
APT_IMPORT_PATTERN_USES_FILESET
APT_MAX_DELIMITED_READ_SIZE
APT_PREVIOUS_FINAL_DELIMITER_COMPATIBLE
APT_STRING_PADCHAR

Reporting

APT_DUMP_SCORE
APT_ERROR_CONFIGURATION
APT_MSG_FILELINE
APT_PM_PLAYER_MEMORY
APT_PM_PLAYER_TIMING
APT_RECORD_COUNTS
OSH_DUMP
OSH_ECHO
OSH_EXPLAIN
OSH_PRINT_SCHEMAS

Sorting

APT_NO_SORT_INSERTION
APT_SORT_INSERTION_CHECK_ONLY

Transport Blocks

APT_LATENCY_COEFFICIENT
APT_DEFAULT_TRANSPORT_BLOCK_SIZE
APT_MAX_TRANSPORT_BLOCK_SIZE/ APT_MIN_TRANSPORT_BLOCK_SIZE

Disk I/O

APT_BUFFER_DISK_WRITE_INCREMENT
APT_CONSISTENT_BUFFERIO_SIZE
APT_EXPORT_FLUSH_COUNT
APT_IO_MAP/APT_IO_NOMAP and APT_BUFFERIO_MAP/APT_BUFFERIO_NOMAP
APT_PHYSICAL_DATASET_BLOCK_SIZE

Decimal Support

APT_DECIMAL_INTERM_PRECISION
APT_DECIMAL_INTERM_SCALE
APT_DECIMAL_INTERM_ROUND_MODE

Building Custom Stages

DS_OPERATOR_BUILDOP_DIR
OSH_BUILDOP_CODE
OSH_BUILDOP_HEADER
OSH_BUILDOP_OBJECT
OSH_BUILDOP_XLC_BIN
OSH_CBUILDOP_XLC_BIN

DB2 Support

APT_DB2INSTANCE_HOME
APT_DB2READ_LOCK_TABLE
APT_DBNAME
APT_RDBMS_COMMIT_ROWS
DB2DBDFT

Look Up support

APT_LUTCREATE_NO_MMAP

Miscellaneous

APT_COPY_TRANSFORM_OPERATOR
APT_EBCDIC_VERSION
APT_DATE_CENTURY_BREAK_YEAR
APT_IMPEXP_ALLOW_ZERO_LENGTH_FIXED_NULL
APT_IMPORT_REJECT_STRING_FIELD_OVERRUNS
APT_INSERT_COPY_BEFORE_MODIFY
APT_ISVALID_BACKCOMPAT
APT_OLD_BOUNDED_LENGTH
APT_OPERATOR_REGISTRY_PATH
APT_PM_NO_SHARED_MEMORY
APT_PM_NO_NAMED_PIPES
APT_PM_SOFT_KILL_WAIT
APT_PM_STARTUP_CONCURRENCY
APT_RECORD_COUNTS
APT_SAVE_SCORE
APT_SHOW_COMPONENT_CALLS
APT_STACK_TRACE
APT_TRANSFORM_COMPILE_OLD_NULL_HANDLING
APT_TRANSFORM_LOOP_WARNING_THRESHOLD
APT_WRITE_DS_VERSION
OSH_PRELOAD_LIBS

Network

APT_IO_MAXIMUM_OUTSTANDING
APT_IOMGR_CONNECT_ATTEMPTS
APT_PM_CONDUCTOR_HOSTNAME
APT_PM_NO_TCPIP
APT_PM_NODE_TIMEOUT
APT_PM_SHOWRSH
APT_PM_STARTUP_PORT
APT_PM_USE_RSH_LOCALLY
APT_RECVBUFSIZE
APT_USE_IPV4
NLS APT_COLLATION_SEQUENCE
APT_COLLATION_STRENGTH
APT_ENGLISH_MESSAGES
APT_IMPEXP_CHARSET
APT_INPUT_CHARSET
APT_OS_CHARSET
APT_OUTPUT_CHARSET
APT_STRING_CHARSET

Oracle Support

APT_ORACLE_LOAD_OPTIONS
APT_ORACLE_NO_OPS
APT_ORACLE_PRESERVE_BLANKS
APT_ORA_IGNORE_CONFIG_FILE_PARALLELISM
APT_ORA_WRITE_FILES
APT_ORAUPSERT_COMMIT_ROW_INTERVAL APT_ORAUPSERT_COMMIT_TIME_INTERVAL

SAS Support

APT_HASH_TO_SASHASH
APT_NO_SASOUT_INSERT
APT_NO_SAS_TRANSFORMS
APT_SAS_ACCEPT_ERROR
APT_SAS_CHARSET
APT_SAS_CHARSET_ABORT
APT_SAS_COMMAND
APT_SASINT_COMMAND
APT_SAS_DEBUG
APT_SAS_DEBUG_IO
APT_SAS_DEBUG_LEVEL
APT_SAS_DEBUG_VERBOSE
APT_SAS_NO_PSDS_USTRING
APT_SAS_S_ARGUMENT
APT_SAS_SCHEMASOURCE_DUMP
APT_SAS_SHOW_INFO
APT_SAS_TRUNCATION

Teradata Support

APT_TERA_64K_BUFFERS
APT_TERA_NO_ERR_CLEANUP
APT_TERA_NO_PERM_CHECKS
APT_TERA_NO_SQL_CONVERSION
APT_TERA_SYNC_DATABASE
APT_TERA_SYNC_USER

Notable new Key Features in DataStage 8.5


Notable new Key Features in DataStage 8.5

DataStage 8.5 is out and IBM has made some significant improvements this time around. Let’s see some of the important enhancements in the new DataStage 8.5 version
  • Its Fast!
DataStage 8.5 is considerably faster than its previous version (8.1). Tasks like saving, renaming, compiling are faster by nearly 40%. The run time performance of jobs has also improved.
  
  • The parallel engine
on DataStage has been tuned to improve performance and resource usage has reduced by 5% when compared to DataStage 8.1

  • XML data
DataStage has historically been inefficient at handling XML files, but in 8.5 IBM has given us a great XML processing package. DataStage 8.5 can now process large XML files (over 30 GB) with ease. Also, we can now process XML data in parallel.
The new XML transform stage can data from multiple sources into a single XML output stream. If you think that is cool, it can also do it the other way around i.e., multiple XML input to a single output stream.
It can also convert data from one XML format to another.

  • Transformer Stage
It is one of the most used and the most important stages on DataStage and it just got better in 8.5
a.     Transformer Looping:
Over the years DataStage programmers have been using workarounds to implement this concept. Now IBM has included it directly in the transformer stage.
There are two types of looping’s available
Output looping: Where we can output multiple output links for a single input link
Ex:
Input Record:
Salesman_name
City_1
City_2
City_3
Jason Bourne
New York
Madrid
New Delhi

Output Record:
Salesman_name
City
Jason Bourne
New York
Jason Bourne
Madrid
Jason Bourne
New Delhi

This is achieved using a new system variable @ITERATION
Input looping: We can now aggregate input records within the transformer and assign the aggregated data to the original input link while sending it to the output.

b.    Transformer change detection:
SaveInputRecord() – Save a record to be used for later transformations within the job
GetInputRecord() – Retrieve the saved record as when it is required for comparisons

c.     System Variables:
                              i.        @ITERATION: Used in the looping mechanism
                            ii.        LastRow(): Indicates the last row in the job
                           iii.        LastRowInGroup(): Will return the last row in the group based on the key column

d.    New NULL Handling features:
In DataStage 8.5 we need not explicitly handle NULL values. Record dropping is arrested if the target column is nullable. We need not handle NULL values explicitly when using functions over columns that have NULL values. And also stage variables are now nullable by default.
APT_TRANSFORM_COMPILE_OLD_NULL_HANDLING has been prepared to support backward compatibility

e.     New Data functions:
There are a host of new date functions incorporated into DataStage 8.5. I personally found the below function most useful
DataFromComponents(years, months, daysofmonth)
Ex: DataFromComponenets(2012,07,20) will output 2012-07-20

DataOffsetByComponents(basedate, years offset, month offset, daysofmonth offset)
Ex: DataOffsetByComponents(2012-07-20, 2,1,1) will output 2014-08-21
DataOffsetByComponents(2012-07-20, -4,0,0) will output 2008-07-20
I will write another detailed blog on the new data functions shortly
 
  • Functionality Enhancements:
-       Mask encryption for before and after job subroutines
-       Ability to copy permissions from one project to a new project
-       Improvements in the multi-client manager
-       New audit tracing and enhanced exception dialog
-       Enhanced project creation failure details

  • Vertical Pivoting:
At long last vertical pivoting has been added
 
  • Integration with CVS
Now in DataStage 8.5 we have the feature that integrates directly with version control systems like CVS. We can now Check-in and Check-out directly from DataStage
 
  • Information Architecture Diagraming Tool:
Now solution architects can draw detailed integration solution plans for data warehouses from within DataStage
 
  • Balanced Optimizer:
As you all know DataStage is an ETL tool. But now with Balanced Optimizer directly being integrated we have the ELT (Extract Load and Transform) feature.
With this we can extract  the data, load it and perform the transformations inside the database engine.

Unix Datastage Interview Questions

Unix Datastage Interview Questions:

1. How to display the 10th line of a file?
head -10 filename | tail -1
2. How to remove the header from a file?
sed -i '1 d' filename
3. How to remove the footer from a file?
sed -i '$ d' filename
4. Write a command to find the length of a line in a file?
The below command can be used to get a line from a file.
sed –n '<n> p' filename
We will see how to find the length of 10th line in a file
sed -n '10 p' filename|wc -c
5. How to get the nth word of a line in Unix?
cut –f<n> -d' '
6. How to reverse a string in unix?
echo "java" | rev
7. How to get the last word from a line in Unix file?
echo "unix is good" | rev | cut -f1 -d' ' | rev
8. How to replace the n-th line in a file with a new line in Unix?
sed -i'' '10 d' filename # d stands for delete
sed -i'' '10 i new inserted line' filename # i stands for insert
9. How to check if the last command was successful in Unix?
echo $?
10. Write command to list all the links from a directory?
ls -lrt | grep "^l"
11. How will you find which operating system your system is running on in UNIX?
uname -a
12. Create a read-only file in your home directory?
touch file; chmod 400 file
13. How do you see command line history in UNIX?
The 'history' command can be used to get the list of commands that we are executed.
14. How to display the first 20 lines of a file?
By default, the head command displays the first 10 lines from a file. If we change the option of head, then we can display as many lines as we want.
head -20 filename
An alternative solution is using the sed command
sed '21,$ d' filename
The d option here deletes the lines from 21 to the end of the file
15. Write a command to print the last line of a file?
The tail command can be used to display the last lines from a file.
tail -1 filename
Alternative solutions are:
sed -n '$ p' filename
awk 'END{print $0}' filename
16. How do you rename the files in a directory with _new as suffix?
ls -lrt|grep '^-'| awk '{print "mv "$9" "$9".new"}' | sh
17. Write a command to convert a string from lower case to upper case?
echo "apple" | tr [a-z] [A-Z]
18. Write a command to convert a string to Initcap.
echo apple | awk '{print toupper(substr($1,1,1)) tolower(substr($1,2))}'
19. Write a command to redirect the output of date command to multiple files?
The tee command writes the output to multiple files and also displays the output on the terminal.
date | tee -a file1 file2 file3
20. How do you list the hidden files in current directory?
ls -a | grep '^\.'
21. List out some of the Hot Keys available in bash shell?
Ctrl+l - Clears the Screen.
Ctrl+r - Does a search in previously given commands in shell.
Ctrl+u - Clears the typing before the hotkey.
Ctrl+a - Places cursor at the beginning of the command at shell.
Ctrl+e - Places cursor at the end of the command at shell.
Ctrl+d - Kills the shell.
Ctrl+z - Places the currently running process into background.
22. How do you make an existing file empty?
cat /dev/null > filename
23. How do you remove the first number on 10th line in file?
sed '10 s/[0-9][0-9]*//' < filename
24. What is the difference between join -v and join -a?
join -v : outputs only matched lines between two files.
join -a : In addition to the matched lines, this will output unmatched lines also.
25. How do you display from the 5th character to the end of the line from a file?
cut -c 5- filename
26. Display all the files in current directory sorted by size?
ls -l | grep '^-' | awk '{print $5,$9}' |sort -n|awk '{print $2}'
Write a command to search for the file 'map' in the current directory?
find -name map -type f
 How to display the first 10 characters from each line of a file?
cut -c -10 filename
Write a command to remove the first number on all lines that start with "@"?
sed '\,^@, s/[0-9][0-9]*//' < filename
How to print the file names in a directory that has the word "term"?
grep -l term *
The '-l' option make the grep command to print only the filename without printing the content of the file. As soon as the grep command finds the pattern in a file, it prints the pattern and stops searching other lines in the file.
How to run awk command specified in a file?
awk -f filename
How do you display the calendar for the month march in the year 1985?
The cal command can be used to display the current month calendar. You can pass the month and year as arguments to display the required year, month combination calendar.
cal 03 1985
This will display the calendar for the March month and year 1985.
Write a command to find the total number of lines in a file?
wc -l filename
Other ways to pring the total number of lines are
awk 'BEGIN {sum=0} {sum=sum+1} END {print sum}' filename
awk 'END{print NR}' filename
How to duplicate empty lines in a file?
sed '/^$/ p' < filename
Explain iostat, vmstat and netstat?
Iostat: reports on terminal, disk and tape I/O activity.
Vmstat: reports on virtual memory statistics for processes, disk, tape and CPU activity.
Netstat: reports on the contents of network data structures.
27. How do you write the contents of 3 files into a single file?
cat file1 file2 file3 > file
28. How to display the fields in a text file in reverse order?
awk 'BEGIN {ORS=""} { for(i=NF;i>0;i--) print $i," "; print "\n"}' filename
29. Write a command to find the sum of bytes (size of file) of all files in a directory.
ls -l | grep '^-'| awk 'BEGIN {sum=0} {sum = sum + $5} END {print sum}'
30. Write a command to print the lines which end with the word "end"?
grep 'end$' filename
The '$' symbol specifies the grep command to search for the pattern at the end of the line.
31. Write a command to select only those lines containing "july" as a whole word?
grep -w july filename
The '-w' option makes the grep command to search for exact whole words. If the specified pattern is found in a string, then it is not considered as a whole word. For example: In the string "mikejulymak", the pattern "july" is found. However "july" is not a whole word in that string.
32. How to remove the first 10 lines from a file?
sed '1,10 d' < filename
33. Write a command to duplicate each line in a file?
sed 'p' < filename
34. How to extract the username from 'who am i' comamnd?
who am i | cut -f1 -d' '
35. Write a command to list the files in '/usr' directory that start with 'ch' and then display the number of lines in each file?
wc -l /usr/ch*
Another way is
find /usr -name 'ch*' -type f -exec wc -l {} \;
36. How to remove blank lines in a file ?
grep -v ‘^$’ filename > new_filename

37. How to display the processes that were run by your user name ?
ps -aef | grep <user_name>
38. Write a command to display all the files recursively with path under current directory?
find . -depth -print
39. Display zero byte size files in the current directory?
find -size 0 -type f
40. Write a command to display the third and fifth character from each line of a file?
cut -c 3,5 filename
41. Write a command to print the fields from 10th to the end of the line. The fields in the line are delimited by a comma?
cut -d',' -f10- filename
42 How to replace the word "Gun" with "Pen" in the first 100 lines of a file?
sed '1,00 s/Gun/Pen/' < filename
43. Write a Unix command to display the lines in a file that do not contain the word "RAM"?
grep -v RAM filename
The '-v' option tells the grep to print the lines that do not contain the specified pattern.
44 How to print the squares of numbers from 1 to 10 using awk command
awk 'BEGIN { for(i=1;i<=10;i++) {print "square of",i,"is",i*i;}}'
45. Write a command to display the files in the directory by file size?
ls -l | grep '^-' |sort -nr -k 5
46. How to find out the usage of the CPU by the processes?
The top utility can be used to display the CPU usage by the processes.
47. Write a command to remove the prefix of the string ending with '/'.
The basename utility deletes any prefix ending in /. The usage is mentioned below:
basename /usr/local/bin/file
This will display only file
48. How to display zero byte size files?
ls -l | grep '^-' | awk '/^-/ {if ($5 !=0 ) print $9 }'
49. How to replace the second occurrence of the word "bat" with "ball" in a file?
sed 's/bat/ball/2' < filename
50. How to remove all the occurrences of the word "jhon" except the first one in a line with in the entire file?
sed 's/jhon//2g' < filename
51. How to replace the word "lite" with "light" from 100th line to last line in a file?
sed '100,$ s/lite/light/' < filename
52. How to list the files that are accessed 5 days ago in the current directory?
find -atime 5 -type f
53. How to list the files that were modified 5 days ago in the current directory?
find -mtime 5 -type f
54. How to list the files whose status is changed 5 days ago in the current directory?
find -ctime 5 -type f
55. How to replace the character '/' with ',' in a file?
sed 's/\//,/' < filename
sed 's|/|,|' < filename
56. Write a command to find the number of files in a directory.
ls -l|grep '^-'|wc -l
57. Write a command to display your name 100 times.
The Yes utility can be used to repeatedly output a line with the specified string or 'y'.
yes <your_name> | head -100
58. Write a command to display the first 10 characters from each line of a file?
cut -c -10 filename
59. The fields in each line are delimited by comma. Write a command to display third field from each line of a file?
cut -d',' -f2 filename
60. Write a command to print the fields from 10 to 20 from each line of a file?
cut -d',' -f10-20 filename
61. Write a command to print the first 5 fields from each line?
cut -d',' -f-5 filename
62. By default the cut command displays the entire line if there is no delimiter in it. Which cut option is used to supress these kind of lines?
The -s option is used to supress the lines that do not contain the delimiter.
63. Write a command to replace the word "bad" with "good" in file?
sed s/bad/good/ < filename
64. Write a command to replace the word "bad" with "good" globally in a file?
sed s/bad/good/g < filename
65. Write a command to replace the word "apple" with "(apple)" in a file?
sed s/apple/(&)/ < filename
66. Write a command to switch the two consecutive words "apple" and "mango" in a file?
sed 's/\(apple\) \(mango\)/\2 \1/' < filename
67. Write a command to display the characters from 10 to 20 from each line of a file?
cut -c 10-20 filename
68. Write a command to print the lines that has the the pattern "july" in all the files in a particular directory?
grep july *
This will print all the lines in all files that contain the word “july” along with the file name. If any of the files contain words like "JULY" or "July", the above command would not print those lines.
69. Write a command to print the lines that has the word "july" in all the files in a directory and also suppress the filename in the output.
grep -h july *
70. Write a command to print the lines that has the word "july" while ignoring the case.
grep -i july *
The option i make the grep command to treat the pattern as case insensitive.
71. When you use a single file as input to the grep command to search for a pattern, it won't print the filename in the output. Now write a grep command to print the filename in the output without using the '-H' option.
grep pattern filename /dev/null
The /dev/null or null device is special file that discards the data written to it. So, the /dev/null is always an empty file.
Another way to print the filename is using the '-H' option. The grep command for this is
grep -H pattern filename
72. Write a command to print the file names in a directory that does not contain the word "july"?
grep -L july *
The '-L' option makes the grep command to print the filenames that do not contain the specified pattern.
73. Write a command to print the line numbers along with the line that has the word "july"?
grep -n july filename
The '-n' option is used to print the line numbers in a file. The line numbers start from 1
74. Write a command to print the lines that starts with the word "start"?
grep '^start' filename
The '^' symbol specifies the grep command to search for the pattern at the start of the line.
75. In the text file, some lines are delimited by colon and some are delimited by space. Write a command to print the third field of each line.
awk '{ if( $0 ~ /:/ ) { FS=":"; } else { FS =" "; } print $3 }' filename
76. Write a command to print the line number before each line?
awk '{print NR, $0}' filename
77. Write a command to print the second and third line of a file without using NR.
awk 'BEGIN {RS="";FS="\n"} {print $2,$3}' filename
78. How to create an alias for the complex command and remove the alias?
The alias utility is used to create the alias for a command. The below command creates alias for ps -aef command.
alias pg='ps -aef'
If you use pg, it will work the same way as ps -aef.
To remove the alias simply use the unalias command as
unalias pg
79. Write a command to display todays date in the format of 'yyyy-mm-dd'?
The date command can be used to display todays date with time
date '+%Y-%m-%d'
------------------------------------------------------------------------------------------------------


1)  Convert single column to single row:
Input: filename : try
REF_PERIOD
PERIOD_NAME
ACCOUNT_VALUE
CDR_CODE
PRODUCT
PROJECT
SEGMENT_CODE
PARTNER
ORIGIN
BILLING_ACCRUAL
Output:
REF_PERIOD PERIOD_NAME ACCOUNT_VALUE CDR_CODE PRODUCT PROJECT SEGMENT_CODE PARTNER ORIGIN BILLING_ACCRUAL
Command: cat try | awk ‘{printf “%s “,$1}’

2) Print the list of employees in Technology department :
Now department name is available as a fourth field, so need to check if $4 matches with the string “Technology”, if yes print the line.
Command: $ awk ‘$4 ~/Technology/’ employee.txt
200  Jason   Developer  Technology  $5,500
300  Sanjay  Sysadmin   Technology  $7,000
500  Randy   DBA        Technology  $6,000
Operator ~ is for comparing with the regular expressions. If it matches the default action i.e print whole line will be  performed.

3) Convert single column to multiple column :
For eg: Input file contain single column with 84 rows then output should be single column data converted to multiple of 12 columns i.e. 12 column * 7 rows with field separtor (fs ;)
Script:
#!/bin/sh
rows=`cat input_file | wc -l`
 cols=12
 fs=;
awk -v r=$rows -v c=$cols -v t=$fs '
 NR<r*c{printf("%s",NR%c?$0"$":$0"\n");next}{print}
 END{if(NR%c&&NR<r*c){print ""}}' input_file > output_file

4) Last field print:
input:
a=/Data/Files/201-2011.csv
output:
201-2011.csv
Command: echo $a | awk -F/ ‘{print $NF}’

5) Count no. of fields in file:
file1: a, b, c, d, 1, 2, man, fruit
Command: cat file1 | awk ‘BEGIN{FS=”,”};{print NF}’
and you will get the output as:8

6) Find ip address in unix server:
Command: grep -i your_hostname /etc/hosts

7) Replace the word corresponding to search pattern:
 >cat file 
 the black cat was chased by the brown dog.
 the black cat was not chased by the brown dog.
 >sed -e '/not/s/black/white/g' file 
 the black cat was chased by the brown dog. 
 the white cat was not chased by the brown dog.

8) The below i have shown the demo for the “A” and “65?.
Ascii value of character: It can be done in 2 ways:
1. printf “%d” “‘A”
2. echo “A” | tr -d “\n” | od -An -t dC
Character value from Ascii:  awk -v char=65 ‘BEGIN { printf “%c\n”, char; exit }’
———————————————————————————————————
9) Input file:
crmplp1 cmis461 No Online
cmis462 No Offline
crmplp2 cmis462 No Online
cmis463 No Offline
crmplp3 cmis463 No Online
cmis461 No Offline
Output –>crmplp1 cmis461 No Online cmis462 No Offline
crmplp2 cmis462 No Online cmis463 No Offline
Command:
awk ‘NR%2?ORS=FS:ORS=RS’ file
———————————————————————————————————
10) Variable can used in AWK
awk -F”$c” -v var=”$c” ‘{print $1var$2}’ filename
———————————————————————————————————
11) Search pattern and use special character in sed command:
sed -e ‘/COMAttachJob/s#”)#.”:JobID)#g’ input_file———————————————————————————————————
12) Get the content between two patterns:sed -n ‘/CREATE TABLE table/,/MONITORING/p’ table_Script.sql———————————————————————————————————
13) Pring debugging script output in log file Add following command in script:
exec 1>> logfilename
exec 2>>logfilename———————————————————————————————————
14) Check Sql connection:#!/bin/sh
ID=abc
PASSWD=avd
DB=sdf
exit | sqlplus -s -l $ID/$PASSWD@$DB
echo variable:$?
exit | sqlplus -s -L avd/df@dfg > /dev/null
echo variable_crr: $?———————————————————————————————————
15) Trim the spaces using sed command

echo “$var” | sed -e ‘s/^[[:space:]]*//’ -e ‘s/[[:space:]]*$//’
Another option is:
Code:
var=$(echo “$var” | sed -e ‘s/^[[:space:]]*//’ -e ‘s/[[:space:]]*$//’)
echo “Start $var End”———————————————————————————————————
16) How to add sigle quote in statement using awk:Input:
/Admin/script.sh abc 2011/08                        29/02/2012 00:00:00
/Admin/script.sh abc 2011/08                        29/02/2012 00:00:00
command:
cat command.txt | sed -e ‘s/[[:space:]]/ /g’ | awk -F’ ‘ ‘{print \x27?$1,$2,$3?\x27?,”\x27?$4,$5?\x27?}’
output:
‘/Admin/script.sh abc 2011/08' ’29/02/2012 00:00:00'
‘/Admin/script.sh abc 2011/08' ’29/02/2012 00:00:00'

Monday, November 28, 2016

DataStage Training online sessions will start soon @Ganeswarreddy,Ph:+971-586938453

I am Going to Start DataStage Training online sessions will start soon @Ganeswarreddy,Ph:+971-586938453

Send me request to below mail id:
ganeshreddy.dspx@gmail.com

Tuesday, August 23, 2016

Netezza cheat sheets

To find database size in Netezza

1SELECT ORX.database::nvarchar(64) AS"DatabaseName",
2case when sum(SOD.allocated_bytes)is null then elseSUM(SOD.allocated_bytes)/1073741824end AS "AllocatedSpace_GB"
3FROM _V_SYS_OBJECT_DSLICE_INFO SODINNER JOIN _V_OBJ_RELATION_XDB ORXON ORX.objid = SOD.tblid
4GROUP BY "DatabaseName"
5ORDER BY "DatabaseName"
-- how-to create a database 
CREATE DATABASE DatabaseName
;
-- how-to rename a database 
ALTER DATABASE OldDatabaseName RENAME TO NewDatabaseName 
;
-- how-to create a synomim 
CREATE SYNONYM synonym_name FOR DatabaseName.SchemaName.TableName 
;
-- how-to create a table 
 
CREATE TABLE SVOC_OWNER.ExampleTable
(
    ByteIntCol        byteint            NOT NULL    
  , SmallIntCol       smallint           NOT NULL    
  , IntegerCol        integer            NOT NULL    
  , BigIntCol         bigint             NOT NULL    
  , NumericPSCol      numeric(38,38)     NOT NULL    
  , NumericPCol       numeric(38,0)      NOT NULL    
  , NumericCol        numeric            NOT NULL    
  , DecimalCol        numeric            NOT NULL    
  , FloatCol          float(15)          NOT NULL    
  , RealCol           real               NOT NULL    
  , DoubleCol         double             NOT NULL    
  , CharCol           char(1)        NOT NULL    
  , VarcharCol        varchar(1)     NOT NULL    
  , NcharCol          nchar(1)       NOT NULL    
  , NvarcharCol       nvarchar(1)    NOT NULL    
  , BooleanCol        boolean            NOT NULL    
  , DateCol           date               NOT NULL    
  , TimeCol           time               NOT NULL    
  , TimeTzCol         timetz             NOT NULL    
  , TimestampCol      timestamp          NOT NULL    
 
 )
DISTRIBUTE ON RANDOM
;
 
-- how-to copy table 
CREATE TABLE NewTable AS SELECT * FROM TableToCopy ; 
 
 
-- how-to or insert data from non-current db to current db table
INSERT INTO TableName SELECT * FROM DatabaseName..TableName
;
-- how-to drop a table 
DROP TABLE DatabaseName..TableName 
; 
-- how-to change the ownership of a table 
ALTER TABLE TableName OWNER TO NewOwner 
;
-- how-to perform a simple select 
SELECT * FROM TableName 
WHERE 
AND 1=1 
AND WhereColumnName = 'WhereCondition' 
AND GreaterThanColumnName > 0.0
ORDER BY WhereColumnName
;
-- how-to delete from table 
DELETE FROM TableNameToDeleteFrom 
WHERE FilterColumnName = 'FilterValue'
;
-- how-to call a stored procedure 
CALL ProcName ; 
EXEC ProcName ; 
EXECUTE ProcName ; 
-- example stored procedure 
CREATE OR REPLACE PROCEDURE ProcName() 
RETURNS INT4 LANGUAGE NZPLSQL AS
BEGIN_PROC
  DECLARE
    StrVar varchar;
  BEGIN
    StrVar := 'This string is quoted';
  END;
END_PROC
; --END PROC
-- a single line comment
/*
a multi-line comment
*/
-- example proc with parameters
CREATE OR REPLACE PROCEDURE ProcName (int, varchar(ANY)) RETURNS int
LANGUAGE NZPLSQL AS
BEGIN_PROC
  DECLARE
    pId ALIAS FOR $1;
    pName ALIAS FOR $2;
  BEGIN
    INSERT INTO t1 SELECT * FROM t2 WHERE id = pId;
  END; 
END_PROC 
; 
-- Control structure
IF movies.genre = 'd' THEN
  film_genre := 'drama';
ELSIF movies.genre = 'c' THEN
  film_genre := 'comedy';
ELSIF movies.genre = 'a' THEN
  film_genre := 'action';
ELSIF movies.genre = 'n' THEN
  film_genre := 'narrative';
ELSE
-- An uncategorized genre form has been requested.
film_genre := 'Uncategorized';
END IF;
-- how-to list all stored procedures 
SHOW PROCEDURE ALL 
; 
-- how-to document a stored procedure
COMMENT ON PROCEDURE customer() IS 'Author: bsmith
Version: 1.0 Description: A procedure that writes a customer name to
the database log file.';
-- how-to list all stored procedures 
SHOW PROCEDURE ALL 
; 
 
-- how-to document a stored procedure
COMMENT ON PROCEDURE customer() IS 'Author: bsmith
Version: 1.0 Description: A procedure that writes a customer name to
the database log file.';
 
-- how-to convert date str into nzdate
select  to_date(substring(20090731 from 1 for 8),'YYYYMMDD') as NZDATE
 
-- select top 
select a.* from some_schema.some_table a limit 10
 
-- START how to remove duplicates =================================
CREATE TABLE TmpTableDuplicates as
    SELECT col11,col2,col3 from DuplicatesContainingTable 
    where FilterCol = 'FilterValue'
    group by 1,2,3
; 
DELETE FROM DuplicatesContainingTable where FilterCol = 'FilterValue'
; 
INSERT INTO Source_table select * from TmpTableDuplicates
; 
DROP TABLE TmpTableDuplicates
; 
-- STOP how to remove duplicates =================================
 
 
-- Query to get a list of views and thier definitions in a database:
SELECT VIEWNAME,OWNER,CREATEDATE,DEFINITION FROM _V_VIEW WHERE OBJTYPE='VIEW';
 
-- Query to get a list of tables in a database:
SELECT TABLENAME,OWNER,CREATEDATE FROM _V_TABLE WHERE OBJTYPE='TABLE';
 
-- Query to get a list of columns from a table or a view:
SELECT ATTNUM,ATTNAME FROM _V_RELATION_COLUMN WHERE NAME=UPPER('<TABLE NAME>')
 ORDER BY ATTNUM ASC;
 
-- Query to get list of user groups on the box:
SELECT GROUPNAME,OWNER,CREATEDATE,ROWLIMIT,SESSIONTIMEOUT,
 QUERYTIMEOUT,DEF_PRIORITY,MAX_PRIORITY FROM _V_GROUP;
 
-- Query to get list of users and the groups they are in, on the box:
 SELECT GROUPNAME,OWNER,USERNAME FROM _V_GROUPUSERS;
-- (Does not give any LDAP users in this query)
 
--Query to find the number of rows in a table without actually querying the table:
-- (Sometimes needed for some really huge tables of rowcount > 80 Billion)
 
SELECT RELNAME TABLE_NAME,
 CASE
 WHEN RELTUPLES < 0
 THEN ((2^32) * RELREFS) + ((2^32) + RELTUPLES )
 ELSE ((2^32) * RELREFS) + ( RELTUPLES )
 END NUM_ROWS
 FROM
 _T_CLASS,
 _T_OBJECT
 WHERE
 _T_OBJECT.OBJID=_T_CLASS.OID AND
 _T_OBJECT.OBJCLASS=4905  DISPLAY ONLY TABLES
 AND RELNAME = UPPER('<TABLE NAME>')
 ;
--Query to check if any of the SPU's are running slower than the rest:
--  (This actually gives the read-write speed of each SPU that is online)
SELECT HWID, BYTE_COUNT/TOTAL_MSEC
 FROM
 _VT_DISK_TIMING
 ORDER BY 2;
 
--- HOW-TO GET THE LIST OF TABLES AND THIER SKEW AND SIZE:
 SELECT TABLENAME,OBJTYPE,OWNER,CREATEDATE,USED_BYTES,SKEW
 FROM _V_TABLE_ONLY_STORAGE_STAT
 WHERE OBJCLASS = 4905 OR OBJCLASS = 4911
 ORDER BY TABLENAME;
 
-- START SELECT INTO
INSERT INTO DatabaseNameTarget.SchemaNameTarget.TableNameTarget 
SELECT ColumnName1 , ColumnName2
FROM DatabaseNameSource.SchemaNameSource.TableNameSource
;
-- STOP SELECT INTO 
 
-- how-to remove duplicates
delete from TableWithDuplicates 
where rowid not in 
(
  select min(rowid) from TableWithDuplicates 
  group by (DuplicateDefiningCol1 , DuplicateDefiningCol2 , DuplicateDefiningCol3) 
);
 
-- _V_USER : The user view gives information about the users in the netezza system.
select * from _v_user;
 
-- _V_TABLE: The table view contains the list of tables created in the netezza performance system.
select * from _v_table;
 
-- _V_RELATION_COLUMN: The relation column system catalog view contains the columns available in a table.
select * from _v_relation_column;
 
-- _V_TABLE_INDEX: This system catalog contains the information about the indexes created on table. netezza does not support creating indexes on a table as of now.
select * from _v_table_index;
 
-- _V_OBJECTS: Lists the different objects like tables, view, functions etc available in the netezza.
select * from _v_objects;
 
-- what is running currently 
select * from _v_qrystat;
 
-- what has been running lately 
select * from _v_qryhist;
 
 
 
 
 
 
 
-- Use \dt in nzsql session to get the list tables
/*
\dv to get list of views
 \dmv - list of materialized views
 \l - list of databases
 \dg - list of groups
 \du - list of users
 \dpu - permissions set to a user
 \dT - list of datatypes
 \d <tablename> - describes the table
 \act - show current active sessions
 \d - describe table(or view,sequence)
 \dt , \dv , \ds , \de - list tables,views,sequences,temp tables
 \dSt , \dSv - list system tables and views
 \df - list functions
 \l - list databases
 \dT - list data types
 \du - list users
 \dg - list groups
 \dpu - list permissions granted to a user
 \dpg - list permissions granged to a group