Oracle GoldenGate 12c: Troubleshooting and Tuning

Course Overview

Gathering Evidence

• Acquiring confidence using the standard Oracle GoldenGate tools normally used to interact with the software:
• GGSCI
• Error Logs, Process Reports, Discard File, System Logs
• Competently using the tools mentioned above to gather evidence about alleged issues occurring to an Oracle GoldenGate environment

Oracle GoldenGate Troubleshooting Tools

• Accessing knowledge Documents - Mailing lists, Oracle GoldenGate forums, etc.
• Consulting the Documentation
• Getting familiar with the ShowSyntax parameter
• Getting familiar with the logdump utility

Basic and Startup Problems

• Getting familiar with basic problems
• Developing the ability to deal with Extract and Replicat startup issues
• Verifying file names, file permissions, consistency between Extract and Replicat group names and their supporting files
• Verifying consistent naming convention when defining trail files and their reference in the Extract/Replicat parameter files
• Verifying that the correct software version has been installed
• Verifying that the Oracle GoldenGate manager is running, that the group name which fails to start exists and is not misspelt
• Verifying that the the parameter file exists, has the same name as the Extract/Replicat group, and is accessible by Oracle GoldenGate
• Introducing the use of CHECKPARAMS to verify that the parameter syntax is correct

Data Extraction Problems

• Developing the ability to deal with and troubleshoot data extraction issues:Extract is slow - Extract abends - Data pump abends
• Analyzing various causes of a slow Extract startup
• Tracing may be enabled, activity logging may be enabled, many columns listed in tables being replicated
• Introducing the use of the CacheMgr parameter to control the virtual memory and temporary disk space that are available for caching uncommitted transaction data
• In-depth analysis of abending Extracts: is Extract unable to open the transaction logs? - Checking to see if the disk is full -Is an archive log missing?
• In-depth analysis of abending data pumps: Does the data pump abend with error number 509 ("Incompatible record")?Have the source tables been changed? Is the Extract abending due to data manipulation?
• Was the primary Extract or the data pump stopped, re-added, or restarted without resetting the data pump’s read position in the local trail?

Data Replication Problems

• Developing the ability to deal with and troubleshoot data replication issues:

• Analyzing the various causes of a "stuck" Replicat: Does "Stats Replicat" return no statistics? Do successive "Info Replicat" commands indicate that the Replicat read checkpoint is not moving?
• Verifying that the Replicat is reading the trail to which Extract is writing
• Analyzing the reasons why a Replicat abends: Is Replicat unable to locate a trail? Is Replicat unable to open a trail file? Was "Add TranData" issued for each table?

• Missed Transactions
• Mapping and Synchronization Problems
• SQLEXEC, File-Maintenance, and Other Problems
• "Classic" Performance Tuning
• "Integrated" Performance Tuning
• "Integrated" Capture and Delivery Troubleshooting

• Acquiring confidence using the standard Oracle GoldenGate tools normally used to interact with the software:
• GGSCI
• Error Logs, Process Reports, Discard File, System Logs
• Competently using the tools mentioned above to gather evidence about alleged issues occurring to an Oracle GoldenGate environment

Oracle GoldenGate Troubleshooting Tools

• Accessing knowledge Documents - Mailing lists, Oracle GoldenGate forums, etc.
• Consulting the Documentation
• Getting familiar with the ShowSyntax parameter
• Getting familiar with the logdump utility

Basic and Startup Problems

• Getting familiar with basic problems
• Developing the ability to deal with Extract and Replicat startup issues
• Verifying file names, file permissions, consistency between Extract and Replicat group names and their supporting files
• Verifying consistent naming convention when defining trail files and their reference in the Extract/Replicat parameter files
• Verifying that the correct software version has been installed
• Verifying that the Oracle GoldenGate manager is running, that the group name which fails to start exists and is not misspelt
• Verifying that the the parameter file exists, has the same name as the Extract/Replicat group, and is accessible by Oracle GoldenGate
• Introducing the use of CHECKPARAMS to verify that the parameter syntax is correct

Data Extraction Problems

• Developing the ability to deal with and troubleshoot data extraction issues:Extract is slow - Extract abends - Data pump abends
• Analyzing various causes of a slow Extract startup
• Tracing may be enabled, activity logging may be enabled, many columns listed in tables being replicated
• Introducing the use of the CacheMgr parameter to control the virtual memory and temporary disk space that are available for caching uncommitted transaction data
• In-depth analysis of abending Extracts: is Extract unable to open the transaction logs - Checking to see if the disk is full -Is an archive log missing
• In-depth analysis of abending data pumps: Does the data pump abend with error number 509 ("Incompatible record")Have the source tables been changed Is the Extract abending due to data manipulation
• Was the primary Extract or the data pump stopped, re-added, or restarted without resetting the data pump’s read position in the local trail

Data Replication Problems

• Developing the ability to deal with and troubleshoot data replication issues:
• Analyzing the various causes of a "stuck" Replicat: Does "Stats Replicat" return no statistics Do successive "Info Replicat" commands indicate that the Replicat read checkpoint is not moving
• Verifying that the Replicat is reading the trail to which Extract is writing
• Analyzing the reasons why a Replicat abends: Is Replicat unable to locate a trail Is Replicat unable to open a trail file Was "Add TranData" issued for each table

Missed Transactions

• Developing the ability to deal with and troubleshoot missed transactions involving issues such as: Tables that are not properly specified - Missing" Add TranData" command - LOB data not replicating
• Analyzing the reasons why transactions are not being processed: Were the tables included in Table and Map statements Are fully qualified names (owner.table) used for both Table and Map
• 'Single quotes' on object names imply case-sensitive names for some databases (including the Oracle database)
• Are the transactions coming out of the source system properly

Mapping and Synchronization Problems

• Developing the ability to deal with and troubleshoot mapping problems, such us: Fetching from the source data fails - Table and column names not matching between the source and the target
• Analyzing the causes of various mapping problems:Is Extract returning fetch-related errors Was the row containing the data deleted
• Has the undo retention expired
• Is "NoUseLatestVersion" for "FetchOptions" being used
• Do the source and target tables exist in the databases Are they specified correctly in the Table or Map statement Are their names fully qualified (schema.table)
• Do table and column names contain only supported characters
• Are WHERE clauses comparing different data types
• Is KeyCols being used Are both source and target KeyCols being used Corresponding source and target columns must match.

SQLEXEC, File-Maintenance, and Other Problems

• Developing the ability to deal with and troubleshoot :
• SQLEXEC problems - File-maintenance problems - Time differences - Too many open cursors
• File-Maintenance Problems: Discard File Is Not Created - Discard File Is Too Big - Trail File Is Too Big
• Miscellaneous Problems: Time Differences - INFO Command Shows Incorrect Status

"Classic" Performance Tuning

• Identifying causes of slow startups
• Monitoring lag and statistics
• Tuning Manager
• Identifying network and disk bottlenecks
• Tuning "Classic" Extract
• Tuning "Classic" Replicat

"Integrated" Performance Tuning

• Describing architecture and advantages of the new Integrated capture
• Describing architecture and advantages of the new Integrated delivery
• Familiarizing with the new V$ tables holding real time Oracle GoldenGate statistics

"Integrated" Capture and Delivery Troubleshooting

• Familiarizing with Troubleshooting Aids and Tools for Integrated Replication
• Describing Oracle GoldenGate Healthcheck for Integrated Replicat
• Describing the Oracle Database 12c GoldenGate AWR report

Prerequisites

Suggested Prerequisites

• Oracle GoldenGate 12c: Fundamentals for Oracle Ed 1
• Familiarity with the Oracle Database