ATG BCC cleaning orphan assets and cleaning Versioned data

ATG BCC cleaning  orphan assets and cleaning Versioned data 

As there are many orphan  assets (products and SKU) and unwanted versioned content(purging not there earlier ), we got the  task from client to remove the all unwanted orphan skus/proudct
This we achieved in the following way.We called this as BCC baseline.

1)We took the the catA/CatB, then idenity the all orphan Assets( skus and prodyucts) and then removed at  catA/catB  through DB Query.
2)Identify GWP promotion SKU's & Product's for the current environment for excluding in the Orphan product sku.
3)Then create projects at BCC by adding all these orphand asstes(skus and proudcts in multiple projects), then execute the project(after business team approve).

Now Export and Import DB.

While doing,we have to follow the below steps carefullly.
1)We need to communite with team regarding the freeze preiod for BCC.
2)Need to turn off the outbound feed.
3)Business team should stop using BCC.
4)Export the BCC topology
5)Shunt down the BCC instance.
6)Backup the versionFileStore file folder from BCC  instance and delete the orginal versionFileStore file folder from BCC instance of PROD instance
7)Export the data dump of  PUB schema and CATA schema of PROD oracle DB
8)Import the data dump of  PUB and CATA schmea to SIT 2 DB(PUB ==> BASELINE_EXP_BCC_PROD  &  CATA ==> BASELINE_EXP_CATA_PROD)
9)Create Two Empty schema  with specific schema meta data only[DDL only]  one for PUB as "BASELINE_IMP_BCC_PROD", one for CATA as "BASELINE_IMP_CATA_PROD"  in the same SIT 2 DB
10)Export the data dump for only price and inventory tables like dcs_plfol_chld, dcs_child_fol_pl, dcs_gen_fol_pl, dcs_price_level, dcs_price_levels,dcs_complex_price,dcs_price,dcs_price_list,dcs_inventory,dcs_inv_atp from PROD core
11)Import the data dump of price and inventory tables like dcs_plfol_chld, dcs_child_fol_pl, dcs_gen_fol_pl, dcs_price_level, dcs_price_levels,dcs_complex_price,dcs_price,dcs_price_list,dcs_inventory,dcs_inv_atp into "BASELINE_ATGCORE" schema of SIT-2 DB
12)Update the "DISTRIBUTED_LOCK_TIMEOUT" value to 3600 seconds in SIT-2 DB and restart the SIT-2 DB instance for reflecting the updated value. This modification is required for full deployment purpose.
13)Audit the count of records for all user tables from "BASELINE_EXP_BCC_PROD" and "BASELINE_EXP_CATA_PROD" schema
14)Create an XLS and store the count of records from both schema's in the expected format(We will see later this format)
15)Export the insert table script for das_id_generator table from "BASELINE_EXP_BCC_PROD" schema
16) Make sure all BCC instance tables exists in the  "BASELINE_IMP_BCC_PROD" schema after the DBA create new schema with only meta data.
17) Make sure all BCC instance tables exists in the  "BASELINE_IMP_CATA_PROD" schema after the DBA create new schema with only meta data..
18)Execute the grant access SQL script in "BASELINE_IMP_CATA_PROD" schema to grant DML access of required tables to "BASELINE_EXP_BCC_PROD" schema
19)Execute the disable constraints SQL script in "BASELINE_IMP_CATA_PROD" schema
20)Import the insert table script for das_id_generator table data into "BASELINE_IMP_BCC_PROD" schema


Execute the Export/Import Process
1)Configure the BASELINE_EXP_BCC_PROD  & BASELINE_EXP_CATA_PROD schema's in  Build Box using the "MonitoredDataSource" & "FakeXADataSource" classes in the "cata_export" & "pub_export" server local config layer
2)Configure the BASELINE_IMP_BCC_PROD  & BASELINE_IMP_CATA_PROD schema's in  Build Box using the "MonitoredDataSource" & "FakeXADataSource" classes in the  "pub_import" server local config layer
3)Configure the CORE schema of the SIT 2 DB in "cata_export",  "pub_export" & "pub_import" server local config layer
4)Execute the Repository Export process as per the Approach/Release document using "cata_export" & "pub_export" server local config layer
5)Verify the exported Files
6)Execute the Import Process for all the repositories as per the Approach/Release document using "pub_import" server local config layer
7)Execute the product catalog manual check-in process using the "ProductCatalog_Manually_checkin.sql" script
8)Audit the count of records for all user tables from the newly imported data in "BASELINE_IMP_BCC_PROD" and "BASELINE_IMP_CATA_PROD" schema
9)Store the  new count of records from both "BASELINE_IMP_BCC_PROD" and "BASELINE_IMP_CATA_PROD" schema's in the expected format.  Shared to client for review purpose.
10)Create/update the datasource for "BASELINE_IMP_BCC_PROD" and "BASELINE_IMP_CATA_PROD" schema's in SIT weblogic console
11)Shut down the BCC instances of SIT2
12)Convert the existing BCC instance SwitchingDataSource_production and SwitchingDataSource_staging components from switching type to normal datasource by changing the class to atg.nucleus.JNDIReference and adding JNDI name through ATG-Data layer
13)Add the RL related hot fix in the BCC instance EAR and also add/update the RL related component changes in the BCC server local config layer
14)Start only SIT BCC instance and this instance must point to the datasource of  "BASELINE_IMP_BCC_PROD" and "BASELINE_IMP_CATA_PROD" schema's
15)Execute the RL through SIT-BCC instance dyn/admin

16)Validate the newly generated versionFileStore location and publishingFileRepository

Verify the DB Import in SIT 2

1)Shut down the instances of SIT2(eStore,BCC,Aux)
2)Convert the existing e-Store instance SwitchingDataSource component  from switching type to normal datasource by changing the class name and adding JNDI name through ATG-Data layer. It will point to "BASELINE_IMP_CATA_PROD"
3)Change the password of the datasource, if applicable
4)Make sure the weblogic datasources of SIT 2 BCC instance and e-store instance point to new imported schemas.
5)Bring Up the BCC server
6)Verify the BCC logs
7)Bring up the store and Aux servers
8)Verify the Store Aux Logs
9)Verify the eStore Instance
10)Configure the BCC agents
11)Perform basic validation of BCC deployment and also publish empty project
12)Perform BCC Full Deployment
13)Verify the BCC publishing of catalog, content  etc repositories assets after modification
14)verify the BCC deployment of config file system  assets like targeters, scenarios, user segement etc by either creating the new asset or updating the exist asset
15)Execute the enable constraints SQL script in "BASELINE_IMP_CATA_PROD" schema
16)Execute CMS in one of the agent
17)Perform Endeca Indexing in SIT-2 environment
18)Verify the Estore instance and mainly the browse and shop pages

Production CutOver
1)Copy the versionFile store to a predefined location
2)Export the data dump from newly imported "BASELINE_IMP_BCC_PROD" and "BASELINE_IMP_CATA_PROD" schema from SIT2
3)Import the exported data dump from "BASELINE_IMP_BCC_PROD" and "BASELINE_IMP_CATA_PROD" schema of  SIT2 schema into QA DB as PUB2,CATA2,CATB2,PRVCATA2,PRVCATB2
4)Delete the data from DMS related tables, E-PUB related tables,  DSS server ID tables
5)Bring down all the instances of PROD
6)Make the weblogic data source changes to point to the new schemas as above
7)Bring up the store,aux instances etc..
8)Bring up the BCC instances
9)Verify the logs
10)Import the BCC Topology
11)Perform Sample Deployment
12)Perform Stibo Deployment
13)Perform Indexing
14)Verify the Functionality/site
15)Users Testing

Export scripts/commands (from DB  to XML file)


The export scripts are executed via repository.It will generate the xml files and log txt files.
As part of the export commands, we have to provide the output file and log file names also.

And at each exort command level, the count query is mentioned for the primary table to validate the data it self.

CAT-A (10):
-----------
1) /atg/multisite/SiteRepository
2) /atg/seo/SEORepository
3) /atg/userprofiling/PersonalizationRepository
4) /atg/commerce/catalog/ProductCatalog
5) /atg/commerce/claimable/ClaimableRepository
6) /atg/commerce/locations/LocationRepository
7) /com/xx/giftregistry/repo/GiftRegistryRepository
8) /com/xx/newsLetter/NewsSubscriptionRepository
9) /com/xx/repo/BrandNumbersRepository
10) /com/xx/common/content/statictext/repo/ContentRepository



From PUB (6):
-------------
1) /atg/systemconfiguration/SystemConfigurationRepository
2) /atg/dynamo/security/AdminSqlRepository
3) /atg/portal/framework/PortalRepository
4) /atg/userprofiling/InternalProfileRepository
5) /atg/web/viewmapping/ViewMappingRepository
6) /atg/epub/file/PublishingFileRepository

LocationRepository :
--------------------
nohup startSQLRepository -m modules.Commerce -s  cata_export -encoding ISO-8859-1 -repository /atg/commerce/locations/LocationRepository -noTransaction -export all exported_location_dump.xml > exported_location_dump_log.txt &
Primary Table :
select count(*) from dcs_location

SEORepository :
---------------
nohup startSQLRepository -m modules.Commerce -s cata_export -encoding ISO-8859-1 -repository /atg/seo/SEORepository -noTransaction -export all exported_SEO_dump.xml > exported_SEO_dump_log.txt &
Primary Table :
select count(*) from das_seo_tag


ClaimableRepository:
---------------------
nohup startSQLRepository  -m modules.Commerce  -s  cata_export -encoding ISO-8859-1 -repository /atg/commerce/claimable/ClaimableRepository -noTransaction -export all exported_claimable_dump.xml  > exported_claimable_dump_log.txt &
Note :
------
Need to override ClaimableRepository.properties to point the SwitchingDataSource.
Primary Table :
select count(*) from dcspp_claimable
select count(*) from dcspp_cp_folder

ProductCatalog :
----------------
nohup startSQLRepository  -m modules.Commerce -s cata_export -encoding ISO-8859-1 -repository /atg/commerce/catalog/ProductCatalog -noTransaction -export all exported_productCatalog_dump.xml > exported_productCatalog_dump_log.txt &
Duration : 13 hours 15 mins
File Size : 7136205408 MB(7.136205408 GB)


GiftRegistryRepository :
------------------------
nohup startSQLRepository  -m modules.Commerce -s  cata_export -encoding ISO-8859-1 -repository /com/xx/giftregistry/repo/GiftRegistryRepository  -noTransaction -export all exported_giftregistry_dump.xml > exported_giftregistry_dump_log.txt &
Primary Table :
select count(*) from xx_event

NewsSubscriptionRepository :
-----------------------------
nohup startSQLRepository  -m modules.Commerce -s  cata_export -encoding ISO-8859-1 -repository /com/xx/newsLetter/NewsSubscriptionRepository -noTransaction -export all exported_newsSubscription_dump.xml > exported_newsSubscription_dump_log.txt &
Primary Table :
select count(*) from xx_NEWS_LETTER

BrandNumbersRepository :
-------------------------
nohup startSQLRepository  -m modules.Commerce -s  cata_export -encoding ISO-8859-1 -repository /com/xx/repo/BrandNumbersRepository -noTransaction -export all exported_brandnumbers_dump.xml > exported_brandnumbers_dump_log.txt &
Primary Table :
select count(*) from xx_BRAND_NUMBERS

ContentRepository :
--------------------
nohup startSQLRepository  -m modules.Commerce -s  cata_export -encoding ISO-8859-1 -repository /com/xx/common/content/statictext/repo/ContentRepository -noTransaction -export all exported_content_dump.xml > exported_content_dump_log.txt &
Primary Table :
select count(*) from xx_static_text 4964
select count(*) from xx_page 258
select count(*) from xx_site_big_static_text 611
select count(*) from xx_site_label_static_text 16896
select count(*) from xx_site_msg_static_text 2044

PersonalizationRepository :
---------------------------
nohup startSQLRepository  -m DPS  -s  cata_export -encoding  ISO-8859-1 -repository /atg/userprofiling/PersonalizationRepository -noTransaction -export all exported_personalization_dump.xml  > exported_personalization_dump_log.txt &
Note :
------
Need to override PersonalizationRepository.properties to point the SwitchingDataSource.
Primary Table :
select count(*) from dps_seg_list_folder
select count(*) from dps_seg_list

SiteRepository :
----------------
nohup startSQLRepository  -m modules.Commerce -s cata_export -encoding  ISO-8859-1 -repository /atg/multisite/SiteRepository/ -noTransaction -export all exported_site_dump.xml  > exported_site_dump_log.txt &
Primary Table :
select count(*) from site_configuration
select count(*) from site_group

SystemConfigurationRepository :
-------------------------------
nohup startSQLRepository  -m modules.CA -s  pub_export -encoding ISO-8859-1 -repository /atg/systemconfiguration/SystemConfigurationRepository  -noTransaction -export all exported_sys_conf_dump.xml > exported_sys_conf_dump_log.txt &
Primary Table :
select count(*) from das_sys_config

AdminSqlRepository :
--------------------
nohup startSQLRepository  -m modules.CA -s  pub_export -encoding ISO-8859-1 -repository /atg/dynamo/security/AdminSqlRepository  -noTransaction -export all exported_admin_sql_dump.xml > exported_admin_sql_dump_log.txt &
Primary Table :
select count(*) from das_account

PortalRepository :
-------------------
nohup startSQLRepository  -m modules.CA -s  pub_export -encoding ISO-8859-1 -repository /atg/portal/framework/PortalRepository  -noTransaction -export all exported_portal_dump.xml > exported_portal_dump_log.txt &
Note :
Need to modify Manifest.MF file under CA module  :
DafEar.Admin DCS-UI.Versioned BIZUI PubPortlet DCS-UI.SiteAdmin.Versioned SiteAdmin.Versioned DCS.Versioned DCS-UI DAF.Endeca.Index.Versioned DCS.Endeca.Index.Versioned DCS.Endeca.Index.SKUIndexing modules.PARALCommonModule modules.PARALModule modules.PARALMerchModule modules.Feed.Import  modules.Feed.Export modules.Commerce
Primary Table :
select count(*) from paf_folder
select count(*) from paf_device_outputs
select count(*) from paf_display_modes
select count(*) from paf_community
select count(*) from paf_template
select count(*) from paf_page_template
select count(*) from paf_page
select count(*) from paf_layout
select count(*) from paf_style
select count(*) from paf_col_palette
select count(*) from paf_title_template
select count(*) from paf_region_def
select count(*) from paf_region
select count(*) from paf_gear_modes
select count(*) from paf_gear_def
select count(*) from paf_gear
select count(*) from paf_gear_param
select count(*) from paf_base_comm_role
select count(*) from paf_comm_template
select count(*) from paf_ct_folder
select count(*) from paf_ct_page
select count(*) from paf_ct_region
select count(*) from paf_ct_gear
select count(*) from paf_ct_alt_gear

InternalProfileRepository :
----------------------------
nohup startSQLRepository  -m DPS.InternalUsers -s  pub_export -encoding  ISO-8859-1 -repository /atg/userprofiling/InternalProfileRepository -noTransaction -export all exported_internal_profile_dump.xml > exported_internal_profile_dump_log.txt &
Note :
Not exported Items:
-------------------
workbench
workbenchTile
workbenchReport
workbenchReportItem

Primary Table :
select count(*) from dpi_user
select count(*) from dpi_contact_info
select count(*) from dpi_mailing
select count(*) from dpi_mail_batch
select count(*) from dpi_mail_server
select count(*) from dpi_role
select count(*) from dpi_access_right
select count(*) from dpi_organization
select count(*) from dpi_folder
select count(*) from wb_workbench
select count(*) from wb_tile
select count(*) from wb_report
select count(*) from wb_report_item
select count(*) from wb_report_msg
select count(*) from dsi_profile_slot
select count(*) from dsi_ind_scenario
select count(*) from dsi_coll_scenario
select count(*) from dsi_scenario_info
select count(*) from dsi_scen_mig_info
select count(*) from dsi_template_info
select count(*) from dsi_coll_trans
select count(*) from dsi_ind_trans
select count(*) from dsi_deletion
select count(*) from dsi_migration
select count(*) from dsi_xref

ViewMappingRepository :
------------------------
nohup startSQLRepository  -m modules.CA -s  pub_export -encoding ISO-8859-1 -repository /atg/web/viewmapping/ViewMappingRepository -noTransaction -export all exported_viewMapping_dump.xml > exported_viewMapping_dump_log.txt &
Primary Table:
select count(*) from vmap_mode
select count(*) from vmap_fh
select count(*) from vmap_im
select count(*) from vmap_ivm
select count(*) from vmap_pvm
select count(*) from vmap_iv
select count(*) from vmap_pv
select count(*) from vmap_ivattrdef
select count(*) from vmap_pvattrdef
select count(*) from vmap_attrval

Import scripts/commands

The following repositories are to be imported which are export using above scripts in the XML file

CAT-A (10):
-----------
1) /atg/multisite/SiteRepository  (imported as part product catalog import )
2) /atg/seo/SEORepository
3) /atg/userprofiling/PersonalizationRepository
4) /atg/commerce/catalog/ProductCatalog
5) /atg/commerce/claimable/ClaimableRepository
6) /atg/commerce/locations/LocationRepository
7) /com/xx/giftregistry/repo/GiftRegistryRepository
8) /com/xx/newsLetter/NewsSubscriptionRepository
9) /com/xx/repo/BrandNumbersRepository
10) /com/xx/common/content/statictext/repo/ContentRepository


From PUB (6):
-------------
1) /atg/systemconfiguration/SystemConfigurationRepository
2) /atg/dynamo/security/AdminSqlRepository
3) /atg/portal/framework/PortalRepository
4) /atg/userprofiling/InternalProfileRepository
5) /atg/web/viewmapping/ViewMappingRepository
6) /atg/epub/file/PublishingFileRepository


Import Queries To PUB Schema :
----------------------------------

InternalProfileRepository :
---------------------------
nohup startSQLRepository -m DPS.InternalUsers -s  pub_import  -repository /atg/userprofiling/InternalProfileRepository -import  /opt/ATG/ATG10.2/home/bin/Exported_Files_On_1_20_2017/CorrectFiles/PUB/exported_internal_profile_dump.xml   > imported_internal_profile_log.txt &

ProductCatalog :
------------------
nohup startSQLImport -m modules.CA -s  pub_import  -repository  /atg/commerce/catalog/ProductCatalog -file  /opt/ATG/ATG10.2/home/bin/Exported_Files_On_1_20_2017/CorrectFiles/CAT-A/exported_productCatalog_dump.xml  -workspace ProductCatalogImport -nocheckin > imported_productCatalog_log.txt &
[info@2020-12-03 04:28:27.608] created new workspace: workspace:75800

[info@2020-12-03 04:28:27.614] parsing input data

CONTAINER:atg.adapter.gsa.sqlimport.ImportException; SOURCE:com.ctc.wstx.exc.WstxUnexpectedCharException: Illegal character ((CTRL-CHAR, code 26))
 at [row,col {unknown-source}]: [28036853,45]
        at atg.adapter.gsa.sqlimport.ImportParserImpl.parseFirsPass(ImportParserImpl.java:130)
        at atg.adapter.gsa.sqlimport.SQLImporter.doImport(SQLImporter.java:1992)
        at atg.adapter.gsa.sqlimport.SQLImporter.execute(SQLImporter.java:1353)
        at atg.adapter.gsa.sqlimport.SQLImporter.main(SQLImporter.java:1306)
Caused by: com.ctc.wstx.exc.WstxUnexpectedCharException: Illegal character ((CTRL-CHAR, code 26))
 at [row,col {unknown-source}]: [28036853,45]
        at com.ctc.wstx.sr.StreamScanner.throwInvalidSpace(StreamScanner.java:666)
        at com.ctc.wstx.sr.StreamScanner.throwInvalidSpace(StreamScanner.java:651)
        at com.ctc.wstx.sr.BasicStreamReader.readCDataPrimary(BasicStreamReader.java:4226)
        at com.ctc.wstx.sr.BasicStreamReader.nextFromTreeCommentOrCData(BasicStreamReader.java:3285)
        at com.ctc.wstx.sr.BasicStreamReader.nextFromTree(BasicStreamReader.java:2801)
        at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1065)
        at org.codehaus.stax2.ri.Stax2EventReaderImpl.nextEvent(Stax2EventReaderImpl.java:255)
        at atg.adapter.gsa.sqlimport.ImportParserImpl.parseFirsPass(ImportParserImpl.java:112)
        ... 3 more
Solution :
sed -ie 's/few/asd/g' hello.txt

Run below on increasing maxConnection @JTData component at server layer and also

Need to add Modules.CA manifest file with below
ATG-Required: DafEar.Admin DCS-UI.Versioned BIZUI PubPortlet DCS-UI.SiteAdmin.Versioned SiteAdmin.Versioned DCS.Versioned DCS-UI DAF.Endeca.Index.Versioned DCS.Endeca.Index.Versioned DCS.Endeca.Index.SKUIndexing modules.PARALCommonModule modules.PARALModule modules.PARALMerchModule modules.Feed.Import  modules.Feed.Export modules.Commerce


nohup startSQLImport -m modules.CA -s pub_import -repository /atg/commerce/catalog/ProductCatalog -file /opt/ATG/ATG10.2/home/bin/exported_productCatalog_dump_22_1_2017_withRemovedChars.xml -workspace ProductCatalogImport -nocheckin -batchSize 5000 > /opt/ATG/ATG10.2/home/bin/imported_productCatalog_log_siva1.txt &

@163 box

nohup startSQLImport -m modules.CA -s pub_import -repository /atg/commerce/catalog/ProductCatalog -file /opt/ATG/ATG10.2/home/bin/Apparao/exported_productCatalog_dump_22_1_2017_withRemovedChars.xml -workspace ProductCatalogImport1 -nocheckin -batchSize 5000 > /opt/ATG/ATG10.2/home/bin/imported_productCatalog_log_apparao_24_1.txt &


workspace :76800

LocationRepository :
---------------------
./startSQLImport -m modules.CA -s pub_import -repository /atg/commerce/locations/LocationRepository -file exported_location_dump.xml -workspace location_import_workspace
**** Error      Thu Dec 03 04:23:13 IST 2020    1606949593882   /       Unable to set configured property "/com/xx/feed/export/listener/IncrementalPriceItemIndexingListener.incrementalQueueRepository" atg.nucleus.ConfigurationException: Unable to resolve component /atg/search/repository/IncrementalItemQueueRepository
**** Error      Thu Dec 03 04:23:13 IST 2020    1606949593915   /       Unable to set configured property "/atg/search/SynchronizationInvoker.port" atg.nucleus.ConfigurationException: Unable to resolve component /atg/search/Constants.null
**** Error      Thu Dec 03 04:23:13 IST 2020    1606949593916   /       Unable to set configured property "/atg/search/SynchronizationInvoker.host" atg.nucleus.ConfigurationException: Unable to resolve component /atg/search/Constants.null
**** info       Thu Dec 03 04:23:13 IST 2020    1606949593967   /atg/epub/deployment/DeploymentManager  Resolving reference to /atg/deployment/DeploymentManager
[info@2020-12-03 04:23:14.089] created new workspace: workspace:75701

[info@2020-12-03 04:23:14.096] parsing input data

[info@2020-12-03 04:23:14.569] importing 184 items.

[info@2020-12-03 04:23:14.569] checking out versioned items in batch

[info@2020-12-03 04:23:15.599] performing add/update/removes

[info@2020-12-03 04:23:16.825] Phase 2 of 2: 100%

[info@2020-12-03 04:23:16.826] checkIn: workspace:75701

java.lang.AbstractMethodError: atg.adapter.version.VersionRepository.getVersionItemsInLine(Latg/versionmanager/DevelopmentLine;ZZZ)Ljava/util/Iterator;
        at atg.adapter.secure.GenericSecuredMutableVersionRepository.getVersionItemsInLine(GenericSecuredMutableVersionRepository.java:883)
        at atg.versionmanager.impl.DevelopmentLineRepositoryImpl.get(DevelopmentLineRepositoryImpl.java:547)
        at atg.versionmanager.impl.DevelopmentLineRepositoryImpl.getAllAssetVersions(DevelopmentLineRepositoryImpl.java:376)
        at atg.versionmanager.impl.DevelopmentLineRepositoryImpl.getAllAssetVersions(DevelopmentLineRepositoryImpl.java:393)
        at atg.versionmanager.impl.DevelopmentLineRepositoryImpl.getAllAssetVersions(DevelopmentLineRepositoryImpl.java:351)
        at atg.versionmanager.impl.DevelopmentLineRepositoryImpl.getAllAssetVersionsCount(DevelopmentLineRepositoryImpl.java:404)
        at atg.versionmanager.impl.WorkspaceRepositoryImpl.checkInAll(WorkspaceRepositoryImpl.java:1490)
        at atg.adapter.gsa.sqlimport.SQLImporter.checkIn(SQLImporter.java:2512)
        at atg.adapter.gsa.sqlimport.SQLImporter.doImport(SQLImporter.java:2043)
        at atg.adapter.gsa.sqlimport.SQLImporter.execute(SQLImporter.java:1353)
        at atg.adapter.gsa.sqlimport.SQLImporter.main(SQLImporter.java:1306)



Solution: Again done  @ 163 box. So issue.

only media-extranal item port :
nohup startSQLImport -m modules.CA -s  pub_import  -repository  /atg/commerce/catalog/ProductCatalog -file  /opt/ATG/ATG10.2/home/bin/exported_media_external_dump.xml  -workspace Only_MediaItem_Import -nocheckin > Only_MediaItem_Import.txt &

nohup startSQLImport -m modules.CA -s  pub_import  -repository  /atg/commerce/catalog/ProductCatalog -file  /opt/ATG/ATG10.2/home/bin/exported_media1_external_dump.xml  -workspace exported_media1_external_dump -nocheckin > exported_media1_external_dump.txt &

SEORepository :
---------------
nohup startSQLImport -m modules.CA -s pub_import -repository  /atg/seo/SEORepository -file  /opt/ATG/ATG10.2/home/bin/Exported_Files_On_1_20_2017/CorrectFiles/CAT-A/exported_SEO_dump.xml  -workspace SEO_Import > imported_SEO_log.txt &

log:

[info@2020-12-05 04:46:48.034] parsing input data

[info@2020-12-05 04:46:48.323] importing 239 items.

[info@2020-12-05 04:46:48.323] checking out versioned items in batch

[info@2020-12-05 04:46:49.338] performing add/update/removes

[info@2020-12-05 04:46:50.472] Phase 2 of 2: 100%

[info@2020-12-05 04:46:50.472] checkIn: workspace:76100

java.lang.AbstractMethodError: atg.adapter.version.VersionRepository.getVersionItemsInLine(Latg/versionmanager/DevelopmentLine;ZZZ)Ljava/util/Iterator;
        at atg.versionmanager.impl.DevelopmentLineRepositoryImpl.get(DevelopmentLineRepositoryImpl.java:547)
        at atg.versionmanager.impl.DevelopmentLineRepositoryImpl.getAllAssetVersions(DevelopmentLineRepositoryImpl.java:376)
        at atg.versionmanager.impl.DevelopmentLineRepositoryImpl.getAllAssetVersions(DevelopmentLineRepositoryImpl.java:393)
        at atg.versionmanager.impl.DevelopmentLineRepositoryImpl.getAllAssetVersions(DevelopmentLineRepositoryImpl.java:351)
        at atg.versionmanager.impl.DevelopmentLineRepositoryImpl.getAllAssetVersionsCount(DevelopmentLineRepositoryImpl.java:404)
        at atg.versionmanager.impl.WorkspaceRepositoryImpl.checkInAll(WorkspaceRepositoryImpl.java:1490)
        at atg.adapter.gsa.sqlimport.SQLImporter.checkIn(SQLImporter.java:2512)
        at atg.adapter.gsa.sqlimport.SQLImporter.doImport(SQLImporter.java:2043)
        at atg.adapter.gsa.sqlimport.SQLImporter.execute(SQLImporter.java:1353)
        at atg.adapter.gsa.sqlimport.SQLImporter.main(SQLImporter.java:1306)


Solution: Again run at 163 box. Assets are checked in.

SystemConfigurationRepository :
-------------------------------
nohup startSQLRepository -m modules.CA -s  pub_import  -repository /atg/systemconfiguration/SystemConfigurationRepository -import  /opt/ATG/ATG10.2/home/bin/Exported_Files_On_1_20_2017/CorrectFiles/PUB/exported_sys_conf_dump.xml   > imported_system_conf_log.txt &




AdminSqlRepository :
--------------------
nohup startSQLRepository -m modules.CA -s  pub_import  -repository /atg/dynamo/security/AdminSqlRepository -import  /opt/ATG/ATG10.2/home/bin/Exported_Files_On_1_20_2017/CorrectFiles/PUB/exported_admin_sql_dump.xml   > imported_AdminSql_log.txt &


PortalRepository :
------------------
nohup startSQLRepository -m modules.CA -s  pub_import  -repository /atg/portal/framework/PortalRepository -import  /opt/ATG/ATG10.2/home/bin/Exported_Files_On_1_20_2017/CorrectFiles/PUB/exported_portal_dump.xml   > imported_portal_log.txt &



PersonalizationRepository :version Repository, executed first as no checkin
---------------------------
nohup startSQLImport -m modules.CA -s pub_import -repository /atg/userprofiling/PersonalizationRepository -file  /opt/ATG/ATG10.2/home/bin/Exported_Files_On_1_20_2017/CorrectFiles/CAT-A/exported_personalization_dump.xml  -workspace Personalization_Import   > imported_personalization_log.txt &


ViewMappingRepository :
-----------------------
nohup startSQLRepository -m modules.CA -s  pub_import   -repository   /atg/web/viewmapping/ViewMappingRepository -import /opt/ATG/ATG10.2/home/bin/Exported_Files_On_1_20_2017/CorrectFiles/PUB/exported_viewMapping_dump.xml   > imported_viewMapping_log.txt &




ClaimableRepository :
---------------------
nohup startSQLImport -m modules.CA -s  pub_import  -repository    /atg/commerce/claimable/ClaimableRepository -file  /opt/ATG/ATG10.2/home/bin/Apparao/exported_claimable_dump.xml  -workspace Claimable_import > imported_Claimable_log_apparao.txt &


LocationRepository : agin run at 163 box
---------------------
nohup startSQLImport -m modules.CA -s pub_import -repository /atg/commerce/locations/LocationRepository -file /opt/ATG/ATG10.2/home/bin/Apparao/exported_location_dump.xml -workspace location_import_workspace > imported_location_log_apparao.txt &


GiftRegistryRepository :
------------------------
nohup startSQLImport -m modules.CA -s  pub_import  -repository    /com/xx/giftregistry/repo/GiftRegistryRepository  -file  /opt/ATG/ATG10.2/home/bin/Apparao/exported_giftregistry_dump.xml  -workspace GiftRegistry_import > imported_giftRegistry_log_apparao.txt &


NewsSubscriptionRepository :
-----------------------------
nohup startSQLImport -m modules.CA -s  pub_import  -repository     /com/xx/newsLetter/NewsSubscriptionRepository -file  /opt/ATG/ATG10.2/home/bin/Apparao/exported_newsSubscription_dump.xml  -workspace NewsSubscription_import  > imported_newsSubscription_log_apparao.txt &

BrandNumbersRepository :
------------------------
nohup startSQLImport -m modules.CA -s   pub_import -repository    /com/xx/repo/BrandNumbersRepository -file  /opt/ATG/ATG10.2/home/bin/Apparao/exported_brandnumbers_dump.xml  -workspace BrandNumbers_imported  > imported_brandnumber_log_apparao.txt &

ContentRepository :
-------------------
nohup startSQLImport -m modules.CA -s  pub_import  -repository  /com/xx/common/content/statictext/repo/ContentRepository -file  /opt/ATG/ATG10.2/home/bin/Apparao/exported_content_dump.xml  -workspace Content_imported > imported_content_log_apparao.txt &

Managing Non-versioned repositories in BCC .BCC Customization

Managing Non-versioned repositories in BCC.

To Manage non-versioned repositories in BCC, we have to follow the below steps.
1)Setup the Repository in the Publishing Module.
2)Add new link to the left navigation bar in BCC home page.
3)Implement a New List View
4)Register Custom services with ApplicationConfigurationManager

Let us see all the steps in detailed manner.
1)Setup the Repository in the Publishing Module.

Create the required tables in the CORE (Production) schema. Configure the repository component in the Publishing module as shown below. Note that the component is created as an instance of GSARepository class and dataSource is pointing to production.

Sample code.

#/com/repository/test/MySampleRepository   

$class=atg.adapter.gsa.GSARepository
$scope=global
repositoryName=MySampleRepository
XMLToolsFactory=/atg/dynamo/service/xml/XMLToolsFactory
dataSource=/atg/dynamo/service/jdbc/JTDataSource
definitionFiles=/com/repository/test/mySampleRepository.xm


2)Add new link to the left navigation bar in BCC home page.
  The ActivityManager (/atg/bizui/activity/ActivityManager) component configures the left-side navigation panel on the BCC home page. Each section of the panel is defined by an element of the ActivityManager’s activitySources array. New sections can be added to the panel by appending ActivitySource to this array.

  An activity source controls the links on the left nav on the BCC home screen. All activity sources are registered with the ActivityManager component (/atg/bizui/activity/ActivityManager). When rendering the home page, the ActivityManager cycles through all the registered ActivitySource components and displays left navigation links for each of them on the BCC home page.


  OOTB ActivitySource components used in BCC

a. Content Administration – /atg/bizui/activity/PublishingActivitySource
b.Search Administration – /atg/bizui/activity/SearchingActivitySource
c.Personalization – /atg/web/personalization/activity/PersonalizationActivitySource
d.Merchanding – /atg/commerce/web/CommerceActivitySource


Each ActivitySource is associated with a sub-panel containing links to “generic activities”, which use specific URLs, and “workflow activity” links, which always present a page that allows the user to create a CA project that uses a workflow of a specific type. Each link is subject to access control, which is determined using activity configuration settings or workflow ACL settings.

To add a new link to the left navigation bar in BCC home page, follow the three steps, which are explained below:

    1.Define A Custom Activity Source component
    2.Define the Generic Activity Definition File
    3.Register this new custom Activity Source with ActivityManager component


      1.Define A Custom Activity Source component
  Create a component that implement the atg.bizui.activity.ActivitySource interface. Here we use the OOTB atg.bizui.activity.PortalActivitySource class.



 # /com/activity/SampleActivitySource     

$class=atg.bizui.activity.PortalActivitySource   
# The name and description for the entry that appears on the left side of the home page
activitySourceResourceBundle=org.com.webui.WebAppResources
activitySourceNameResource=activitySourceName
activitySourceDescriptionResource=activitySourceDescription

# Generic activity definitions
genericActivityDefinitionFiles=/com/activity/genericActivities.xml
genericActivityFileModificationInterval=every 300 seconds

securityConfiguration=/atg/dynamo/security/SecurityConfiguration
XMLToolsFactory=/atg/dynamo/service/xml/XMLToolsFactory
scheduler=/atg/dynamo/service/Scheduler

defaultCreateWorkflowURL=/atg/bcc/process?successURL=/AssetManager/assetManager.jsp

# Default URL for the main project page.
defaultProcessURL^=/atg/bizui/activity/PublishingActivitySource.defaultProcessURL

# Default URL for the workflow task page. All tasks use the AssetManager UI.
defaultTaskURL=/AssetManager/assetManager.jsp
fallbackPriority=2


2 – Define the Generic Activity Definition File
Now we need to create the genericActivities.xml file which is specified in the genericActivityDefinitionFiles property of the new ActivitySource component.
The genericActivities.xml file specifies the links on the left nav used for browsing repositories.


#/com/activity/genericActivities.xml

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE gsa-template
PUBLIC "-//Art Technology Group, Inc.//DTD General SQL Adapter//EN"
"http://www.atg.com/dtds/activitymanager/activitymanager_1.0.dtd">

<!-- This adds activity links to the "Non Versioned" section of the BCC home page -->
 <generic-activities>
 <!-- Add a link named "Edit My Sample Repository Items" -->
 <activity>
<id>editSampleItems</id>
 <resource-bundle>org.com.webui.WebAppResources</resource-bundle<display-name-resource>editSampleItems.displayName</display-name-resource>
 <description-resource>editSampleItems.description</description-resource>
 <destination-page>
<!-- This link takes the user to the AssetManager UI -->
<url>/AssetManager/assetManager.jsp</url!-- The clear-context flag clears any active Content Administration project references out of the user's session. -->
<clear-context>true</clear-context>
<!-- The ACL is optional. This ACL limits the visibility of this link to people with any of the epub roles. -->
<acl>Profile$role$epubAdmin:read;Profile$role$epubSuperAdmin:read;Profile$role$epubManager:read;Profile$role$epubUser:read</acl>
</destination-page>
</activity>
 </generic-activities>


3 – Register this new Activity Source with ActivityManager

The path of the OOTB component is /atg/bizui/activity/ActivityManager

#/atg/bizui/activity/ActivityManager
 activitySources+=/com/activity/SampleActivitySource


Implement a New List View
To add a new List View in BCC, follow the four steps, which are explained below:

1 – Create a View Configuration
Create a View Configuration component of class FilterableBrowseListViewConfiguration.

#/com/web/assetmanager/SampleItemViewConfiguration.properties

$class=atg.web.assetmanager.configuration.FilterableBrowseListViewConfiguration
   
# The path to the repository from which to draw the items
 repositoryPath=/com/repository/test/MySampleRepository

# The type of items to display
assetTypeName=sampleItem

# what types the user is allowed to create in this view
createableTypesList=sampleItem

# how many items to show per page
itemsPerPage=50



Note:createableTypesList can refers multiple items separated by comma and shown these items as dropdown to create any new items.

We can refer the OOTB configuration for /atg/web/assetmanager/configuration/externalprofile/UserViewConfiguration as an example.


2 – Add the view to the task configuration file.
When the Business Control Center starts, the Task Configuration Manager component locates the default task configuration file, ‘taskConfiguration.xml‘, as well as any custom task configuration files you have created. The Task Configuration Manager parses the combined file, organizing the
information so that when a page is requested, the Business Control Center can determine what UI elements to display based on the current activity, task, or workflow.
In the task configuration file, configure the Tabs and Buttons for the new list view. Insert tags for modifying specified activity, task, or workflow configurations. You can make the following modifications:

a. Add support for new views, item types, or properties.
b. Remove existing tabs, buttons, views, item type, and properties.
c. Modify existing settings, such as the item types supported in the Search tab




# /atg/web/assetmanager/configuration/taskConfiguration.xml

<?xml version="1.0" encoding="ISO-8859-1" ?>
<!DOCTYPE task-configuration
 PUBLIC "-//Art Technology Group, Inc.//DTD Asset Manager Task Configuration//EN"
 'http://www.atg.com/dtds/web/assetmanager/taskConfiguration_1.0.dtd'>
 <task-configuration>
<!-- Define an activity for editing the My Sample repository -->
 <activity id="editSampleItems" inherit-from="assetManager.defaultEdit">
 <activity-name>editSampleItems</activity-name>
 <!-- Configure the buttons on the toolbar -->
<operations>
 <operation>create</operation>
 <operation>delete</operation>
 </operations>
 <!-- Define tabs for the left pane -->
 <tabs>
 <tab-order>
<tab-id>browse</tab-id>
 <tab-id>search</tab-id>
 </tab-order>
 <initial-tab>browse</initial-tab>
 <!-- Configure the Browse tab with a list view of inventory items -->
 <tab id="browse">
 <display-name-resource>assetManager.tab.browse</display-name-resource<page>/browse/browseTab.jsp</page<views>
 <view-order>
<view-id>editSampleItems.sampleItem</view-id>
 </view-order>
 <initial-view>editSampleItems.sampleItem</initial-view>
 <view id="editSampleItems.sampleItem">
 <resource-bundle>org.com.webui.WebAppResources</resource-bundle<display-name-resource>editSampleItem.browseTab.view.sampleItem</display-name-resource<configuration>/com/web/assetmanager/SampleItemViewConfiguration</configuration>
 <page>/browse/list.jsp</page</view>
 </views>
 </tab>
<!-- Configure the Search tab to allow searching for inventory items -->
 <tab id="search">
 <display-name-resource>assetManager.tab.search</display-name-resource<page>/search/searchTab.jsp</page<views>
 <initial-view>form</initial-view>
 <view id="form">
 <page>/search/searchForm.jsp</page<item-types>
 <item-type>/com/repository/test/MySampleRepository:sampleItem</item-type>
 </item-types>
 </view>
 <view id="results">
 <page>/search/searchResults.jsp</page</view>
 </views>
 </tab>
 </tabs>
 </activity>

 </task-configuration>

Note:As part of any tab, we can add either configuration file using configuration  tag or we can directly mention repository and repository items.

<item-types>
  <item-type>/xxx/xxxRepository:xxxevent</item-type>
 </item-types>


And we use the following operation if we have requirements.

 <operations>
           
              <operation combine="remove">link</operation>
              <operation combine="remove">unlink</operation>
            </operations>

Register Custom services with ApplicationConfigurationManager

3 – Create services for the repository Item.


#/com/remote/controlcenter/service/sampleItems.xml   

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
 <!DOCTYPE application-configuration
 PUBLIC "-//Art Technology Group, Inc.//DTD BCC Application Configuration //EN"
 "http://www.atg.com/dtds/application-configuration/application-configuration_1.0.dtd">

<applications>
 <application id="editSampleItems">
 <resource-bundle>org.com.webui.WebAppResources</resource-bundle<display-name-resource>editSampleItems.displayName</display-name-resource>
 <destination-page>
 <url>/AssetManager/assetManager.jsp</url<query-parameter key="project" value="-1" />
 <query-parameter key="activity" value="editSampleItems" />
 </destination-page>
 <acl>Profile$role$epubAdmin:read;Profile$role$epubSuperAdmin:read;Profile$role$epubManager:read;Profile$role$epubUser:read</acl>
 <initializer>
 </initializer>
 <sort-priority>30</sort-priority>
 <category>yourProjectExtras</category>
 </application>
 </applications>

Note: editSampleItems (id in genericActivities.xml  file)So the same id should be used in services sampleItems.xml  file ,genericeActivities.xml  file. and taskConfiguration.xml

4 – Register your custom services to ApplicationConfigurationManager.

The path of the OOTB component is /atg/remote/controlcenter/service/ApplicationConfigurationManager

#/atg/remote/controlcenter/service/ApplicationConfigurationManager

applicationFiles+=/com/remote/controlcenter/service/sampleItems.xml

Now we can see the non versioned repository at BCC.

BCC HOME page left side links

How the BCC home page left navigation and links are displayed?

 activitySources is the property which controls the links on the left navigation on the BCC home screen. All activity sources are registered with the ActivityManager component at /atg/bizui/activity/ActivityManager. When rendering the BCC home page, the ActivityManager cycles through all the registered ActivitySource components and displays left navigation links for each of them on the BCC home page.

 For example if I want to add a new link "My New Link" , we need to follow the below steps.

 1)activityManager
Add  activityManager.properties to specify the activityresources. In this  activityManager, I specified one MyActivitySource.

   atg\bizui\activity\ActivityManager.properties
     activitySources+=\
                       /com/xyzpath/activity/MyActivitySource
 
  2)MyActivitySource.properties   component 

     Add  MyActivitySource.properties  to specify the name of the link and the other details . Here it refers to a bundle properties file.

MyActivitySource.properties
$class=atg.bizui.activity.PortalActivitySource

# The name and description for the activity source entry that appears on the left side of the home page.
activitySourceResourceBundle=my.webui.WebAppResources
activitySourceNameResource=xxRepositoryActivitySource
activitySourceDescriptionResource=xxRepositoryActivitySource

# Generic activity definitions
genericActivityDefinitionFiles=/com/xyzpath/activity/genericActivities.xml
genericActivityFileModificationInterval=every 300 seconds

# URLs for the create-project page, the main project page, and the task page.
defaultCreateWorkflowURL=/atg/bcc/process?successURL=/AssetManager/assetManager.jsp
defaultProcessURL^=/atg/bizui/activity/PublishingActivitySource.defaultProcessURL
defaultTaskURL=/AssetManager/assetManager.jsp

fallbackPriority=2
displayOrderPriority=30
securityConfiguration=/atg/dynamo/security/SecurityConfiguration
XMLToolsFactory=/atg/dynamo/service/xml/XMLToolsFactory
scheduler=/atg/dynamo/service/Scheduler


3) WebAppResources.properties

xxRepositoryActivitySource=Manage xx Repository
xxRepositoryControl.displayName=xx Repository
xxRepositoryControl.description=Manage xx Repository

4)genericActivities.xml
/com/xyzpath/activity/genericActivities.xml
  <?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE generic-activities SYSTEM "dynamosystemresource:/atg/dtds/activitymanager/activitymanager_1.0.dtd">
<generic-activities>
  <activity>
    <id>eventRepositoryControl</id>
    <resource-bundle>my.webui.WebAppResources</resource-bundle>
    <display-name-resource>xxRepositoryControl.displayName</display-name-resource>
<description-resource>xxRepositoryControl.description</description-resource>
    <destination-page>
      <url>/AssetManager/assetManager.jsp</url>
      <clear-context>true</clear-context>
      <acl>Profile$role$eventControl:read</acl>
    </destination-page>
  </activity>
</generic-activities>

HTTP connection via program

HTTP connection via program

Using the below code snippet, We can call/invoke the external URL using GET/POST method.

import org.apache.commons.httpclient.Credentials;
import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.HttpException;
import org.apache.commons.httpclient.SimpleHttpConnectionManager;
import org.apache.commons.httpclient.UsernamePasswordCredentials;
import org.apache.commons.httpclient.auth.AuthScope;
import org.apache.commons.httpclient.methods.GetMethod;
import org.apache.commons.httpclient.methods.PostMethod;
import org.apache.commons.httpclient.params.HttpConnectionManagerParams;



HttpConnectionManagerParams httpParams = new HttpConnectionManagerParams();
SimpleHttpConnectionManager localSimpleHttpConnectionManager = new SimpleHttpConnectionManager();
httpParams.setConnectionTimeout((int) Long.parseLong(getAkamaiCacheConnectionTimeOut()));
localSimpleHttpConnectionManager.setParams(httpParams);
HttpClient localHttpClient = new HttpClient(localSimpleHttpConnectionManager);

// proxy is requried only in local enviroment. Not required at higher environments.
if (isEnableProxyFlag()) {
if (isLoggingDebug()) {
logDebug("Inside proxy check block");
}
//lProxy_host,lProxyPort are specific environment related proxy information.
String lProxy_host = (String) getProxyMap().get("proxyHost");
int lProxyPort = Integer.parseInt((String) getProxyMap().get("proxyPort"));
localHttpClient.getHostConfiguration().setProxy((String) lProxy_host, lProxyPort);
userObject = (String) getProxyMap().get("proxyUserName");
password = (String) getProxyMap().get("proxyPassword");
if ((userObject != null) && (password != null)
&& ("true".equalsIgnoreCase((String) getProxyMap().get("proxyAuthrequired")))) {
responseObject = new UsernamePasswordCredentials((String) userObject, password);
localHttpClient.getState().setProxyCredentials(new AuthScope((String) lProxy_host, lProxyPort),
(Credentials) responseObject);
}
}

try {
String nodeUrl ="some external system URL toinvoked"

postURL = new URL(nodeUrl);
//using the getmethod as URL to be invoked team speficied.It may be post method also. based on requirement.
getMethod = new GetMethod(postURL.toString());
int httpReponseCode = localHttpClient.executeMethod(getMethod);

}} catch (HttpException e1) {
if (isLoggingError()) {
logError(" HttpException occurs");
logError("Not able to connect to url " + nodeUrl);
}
} catch (MalformedURLException e1) {
if (isLoggingError()) {
logError(" MalformedURLException occurs ");
logError("Not able to connect to url " + nodeUrl);
}

} catch (IOException e1) {
if (isLoggingError()) {
logError(" IOException occurs ");
logError("Not able to connect to url " + nodeUrl);
}
}


}

Requried JARs
The following jars are required for this.

lib/org-apache-commons-codec.jar lib/org.apache.commons.httpclient.jar and need to add as part of META-INF/MANIFEST.MF

ATG DeploymentEventListener BCC

ATG DeploymentEventListener

We can add DeploymentEventListener in either following two components.

1)DeploymentServer

2)DeploymentAgent

DeploymentServer:

atg\epub\DeploymentServer.properties

allowMissingNonEssentialAgents=true

deploymentEventListeners+=\

/com/xyzpath/deployment/listener/xxDeploymentEventListener

Sample code of xxDeploymentEventListener
At the Deployment server/CA side listner, can get the project and  from project we can get assets.

public class xxDeploymentEventListener extends GenericService implements DeploymentEventListener
{

public void deploymentEvent(DeploymentEvent pDeploymentevent) {

Map<String, Set<String>> affectedRepositoriesFromThisDeployment = pDeploymentevent.getAffectedItemTypes();
//if deployment is completed
if(pDeploymentevent != null && pDeploymentevent.getNewState() == DeploymentEvent.DEPLOYMENT_COMPLETE) {
if (affectedRepositoriesFromThisDeployment != null && !affectedRepositoriesFromThisDeployment.isEmpty()) {

Iterator<String> iterator = affectedRepositoriesFromThisDeployment.keySet().iterator();
while (iterator.hasNext()) {
String repositoryPath = iterator.next();
if (StringUtils.isNotBlank(repositoryPath) && "/com/xyzpath/componentName".equalsIgnoreCase(repositoryPath)) {
String[] projectIds = pDeploymentevent.getDeploymentProjectIDs();
Project project = null;
Set workSpaceAssets = null;
Iterator<AssetVersion> workspaceAssetsIterator = null;
RepositoryItem promoRepItem = null;

try {
if (projectIds != null && projectIds.length >= CommonConstants.ONE) {
EPubHomes.initialize();
EPubHomes persistentHomes = ProjectConstants.getPersistentHomes();
ProjectHome home = persistentHomes.getProjectHome();
try {
project = home.findById(projectIds[CommonConstants.ZERO]);
} catch (EJBException | FinderException exception) {
vlogError("Cannot find project with Id " + projectIds[CommonConstants.ZERO], exception);
}
}
RepositoryItem repositoryItem =null;
if (project != null) {
workSpaceAssets = project.getAssets();
}
if (workSpaceAssets != null) {
workspaceAssetsIterator = (Iterator<AssetVersion>) workSpaceAssets.iterator();
}

if (workspaceAssetsIterator != null) {
while (workspaceAssetsIterator.hasNext()) {
AssetVersion assetVersion = workspaceAssetsIterator.next();


try {
if (assetVersion != null) {

repositoryItem = assetVersion.getRepositoryItem();
}
}catch (RepositoryException repoExc) {
vlogError("Repository Exception while accessing item " , repoExc);
}

}
}


}catch (VersionException | DataModelException Exc) {

}

}

}

}

}

DeploymentAgent:

atg/epub/DeploymentAgent.properties

switchableDataStores+=/atg/dynamo/service/jdbc/SwitchingDataSource,/atg/epub/file/ConfigFileSystem

deploymentEventListeners+=/com/xyzpath/deployment/listener/yyyDeploymentEventListener


The DeploymentAgent side listener doesnot have Project and can't access the assets from project.


Sample code of yyyDeploymentEventListener

In this case we can verify  only the affected repository path

public class yyyDeploymentEventListener extends GenericService implements DeploymentEventListener {

public void deploymentEvent(DeploymentEvent pDepEvent) {
if(DeploymentEvent.DEPLOYMENT_COMPLETE == pDepEvent.getNewState()){

Map<String,Set<String>> affectedRepositoriesFromThisDeployment = pDepEvent.getAffectedItemTypes();
if(affectedRepositoriesFromThisDeployment != null && !affectedRepositoriesFromThisDeployment.isEmpty()){
String repositoryPathAsKey = null;
Iterator<String> iterator =  affectedRepositoriesFromThisDeployment.keySet().iterator();
while(iterator.hasNext()){
repositoryPathAsKey = iterator.next();
vlogDebug("repositoryPathAsKey :{0}", repositoryPathAsKey);
//checking the repository
if((StringUtils.isNotBlank(repositoryPathAsKey) &"com/xx/zzcompoentPath".equalsIgnoreCase(repositoryPathAsKey)) ||(StringUtils.isNotBlank(repositoryPathAsKey) && repositoryPath.equalsIgnoreCase(repositoryPathAsKey))){
//need to write project requirem,ent logic
}
}
}
}

}


}

ATG Email Configuration

The following steps need to follow for EMAIL implementation.

Need to configure the following values in any component where email logic is written.
1) configuration at component (properties file)level
   templateEmailSender=/atg/userprofiling/email/TemplateEmailSender
  contentProcessor=/atg/userprofiling/email/HtmlContentProcessor
  fillFromTemplate=true

2)Code to sent email as part of java code
  TemplateEmailInfoImpl templateEmailInfo = null;
  templateEmailInfo = new TemplateEmailInfoImpl();


param.put("emailContent",htmlMessage);
templateEmailInfo.setContentProcessor(getContentProcessor());
templateEmailInfo.setFillFromTemplate(isFillFromTemplate());
templateEmailInfo.setTemplateParameters(param);
templateEmailInfo.setTemplateURL("email template jsp path");
templateEmailInfo.setMessageFrom(senderEmail);

String recipientArray[] = {receipeingEmailId};
getTemplateEmailSender()).sendEmailMessage(templateEmailInfo, recipientArray);


3) The JSP or email template should have this code as follows to display the content from param.
<dsp:valueof param="emailContent" valueishtml="true"/>.


4)SMTPEmail component
SMTPEmail.properties

$class=atg.service.email.SMTPEmailSender
# Remote smtp host to actually send the email.email handler server ip address
emailHandlerHostName=xx.yxx.ww
emailHandlerPort=25
defaultFrom=noreply@liverpool.com
waitForConnectionMillis=200004)

5)Need to update the \atg\dynamo\Configuration.properties

siteHttpServerPort=local server port
siteHttpServerName=local server address

updating dynamic value as part of the HTML content using backend (used L--l )

If we want to update the dynamic values  as part of the HTML content, we have to use the following code snippet.If we want to sent different values for different users,then only we have to use this.
The classes Document,Jsoup,Element are available in the jar. jsoup-1.11.2.jar.
We have to add the jar as part lib folder of the particular module and update in MANIFEST.MF file as follows.

ATG-Class-Path: lib/jsoup-1.11.2.jar

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;

Document templateDoc = null;
try{
//the incoming HTML conent which come as part of JSON request.
String htmlContent = pHtmlContent;
templateDoc = Jsoup.parse();
Element xxxTag = templateDoc.getElementById("Id in html document");
if(xxxTag != null){
xxxTag.html("some application dynamic value to do added in the document");
}

}catch(MalformedURLException e){

}

patch bay info or ATG messaging info Dynamo Messaging

We have to change the following files for patch functionality or any asynchronous calls

1)atg\dynamo\messaging\dynamoMessagingSystem.xml
2)\atg\dynamo\messaging\SqlJmsProvider.properties

1)dynamoMessagingSystem.xml

We have to add the Source and sink code as part of this file. Generaaly the source code will be executed at frontend sources.
Sink will be executed at backend/AUX instances.


Sample Souce Code (Generally it will be part of front end servers/services code/module)

<?xml version="1.0" encoding="ISO-8859-1" ?>
<dynamo-message-system>
<patchbay>
<message-source>
<nucleus-name>/com/xxxpath/message/messageSource/XXXMessageSource</nucleus-name>
<output-port>
<port-name>XXXSourceYYY</port-name>
<output-destination>
<provider-name>sqldms</provider-name>
<destination-name>sqldms:/XXXXXQueue</destination-name>
<destination-type>Queue</destination-type>
</output-destination>
</output-port>
</message-source>

</patchbay>
</dynamo-message-system>


Sample Sink Code (Ideally the sink code will be part of the Backend/Aux instances)

<?xml version="1.0" encoding="ISO-8859-1" ?>
<dynamo-message-system>
<patchbay>
<message-sink>
<message-sink>
<nucleus-name>/com/xxx/message/messageSink/xxxMessageSink</nucleus-name>
<input-port>
<port-name>xxxSinkyyy</port-name>
<input-destination>
<provider-name>sqldms</provider-name>
<destination-name>sqldms:/aaaQueue</destination-name>
<destination-type>Queue</destination-type>
<redelivery>
<max-attempts>3</max-attempts>
<delay>60000</delay>
<failure-output-port>xxxError</failure-output-port>
</redelivery>
</input-destination>
</input-port>
<redelivery-port>
<port-name>xxxError</port-name>
<output-destination>
<provider-name>sqldms</provider-name>
<destination-name>sqldms:/xxxxErrorQueue</destination-name>
<destination-type>Queue</destination-type>
</output-destination>
</redelivery-port>
</message-sink>
</patchbay>
</dynamo-message-system>

(Need to add the destination name which mentioned in the dynamo messaging file in the  SqlJmsProvider.properties file)
2)SqlJmsProvider.properties   (in Sounce module)
requiredQueueNames+=\XXXXXQueue,\
                    \yyyQueue1   (some other destination name, not related to this )

  SqlJmsProvider.properties   (in sink module)
requiredQueueNames+=\aaaQueue,\
                    \xxxxErrorQueue

ATG 11.3 JAXRS information

1)Configuration as part of web.xml

<servlet>
    <servlet-name>Jersey</servlet-name>
    <servlet-class>atg.service.jaxrs.JerseyServletWrapper</servlet-class>
    <init-param>
        <param-name>jersey.config.server.provider.packages</param-name>
        <param-value>com.xxx.yyy.resource.abc</param-value>  (might be not required)
    </init-param>
    <init-param>
      <param-name>atg.service.jaxrs.JerseyServletWrapper.servletClasses</param-name>
      <param-value>org.glassfish.jersey.servlet.ServletContainer,atg.service.swagger.SwaggerBootstrap</param-value>
    </init-param>
    <!-- Params for JerseyServletWrapper servlet  -->
    <init-param>
      <param-name>javax.ws.rs.Application</param-name>
      <param-value>atg.service.jaxrs.JAXRSApplication</param-value>
    </init-param>
    <init-param>
      <param-name>jersey.config.server.provider.classnames</param-name>
      <param-value>atg.service.swagger.SwaggerApiListingResource,io.swagger.jaxrs.listing.SwaggerSerializers</param-value>
    </init-param>
    <init-param>
   <!--  Params for JAXRSApplication  -->
      <param-name>atg.service.jaxrs.JAXRSApplication.resourceRegistryPath</param-name>
      <param-value>/atg/dynamo/service/jaxrs/RestResourceRegistry</param-value>
    </init-param>
    <init-param>
      <param-name>atg.service.jaxrs.JAXRSApplication.payloadSchemaRegistryPath</param-name>
      <param-value>/atg/dynamo/service/payloadschema/PayloadSchemaRegistry</param-value>
    </init-param>
    <init-param>
    <!-- Params for SwaggerBootstrap servlet  -->
      <param-name>atg.service.swagger.SwaggerBootstrap.bootstrapService</param-name>
      <param-value>/atg/dynamo/service/swagger/SwaggerBootstrapService</param-value>
    </init-param>
    <load-on-startup>3</load-on-startup>
    <async-supported>true</async-supported>
  </servlet>
  <servlet-mapping>
    <servlet-name>Jersey</servlet-name>
    <url-pattern>/services/*</url-pattern>
  </servlet-mapping>



The following are components


1) /atg/dynamo/service/payloadschema/PayloadSchemaRegistry.properties
   definitionFile=atg/dynamo/service/payloadschema/payloadSchema.xml

2)atg/dynamo/service/payloadschema/payloadSchema.xml


3) atg/dynamo/service/jaxrs/JerseyClassLoaderService.properties
   childFirstPrefixes+=\
                                   com/xxxpath/resource,\
                                   com/yypath/reosurce2
classpathFiles+=\

The childFirstPrefixes should have the paths of the all services.


4)atg/dynamo/service/jaxrs/RestExceptionMapper.properties
    $class=com.Lxx.gyy.p1.p2.xxxExceptionMapper
$classloader=/atg/dynamo/service/jaxrs/JerseyClassLoaderService


5) /atg/dynamo/service/jaxrs/RestResourceRegistry.properties

     nucleusRestResources+=\
                         /com/xxxpath/yyyyService,\
                            /come/zzzpath/absService

The RestResourceRegistry can also register a Nucleus REST resource with Jersey. To register a Nucleus component, you must add it to the RestResourceRegistry.properties file.


The End point service should be as follows.

yyyyService.properties
$class=com.xxxpath.resource.XXXService
$classloader=/atg/dynamo/service/jaxrs/JerseyClassLoaderService
responsePayload=/com/abcPath/response/xxxPayload
xxxManager=/com/xyzpath/manager/XXXManager (regular nucleus components)


XXXService.java(Resource class)

@RestResource(id = "xxxService")
@Path("/xxx")
@Api("xxx")
public class XXXService  extends GenericService{

   @POST
@Path("/function")
@Endpoint(id = "/function#POST", filterId = "xxx-Filter",validatorId="xxxValidator", isSingular = true)
public RepresentationModel function(JSONObject pInputData){

xxxPayload xxxPayLoad=new xxxPayload();
RepresentationModel.Builder builder = null;
builder = new RepresentationModel.Builder().state(xxxPayLoad);
return builder.build();
}
}

Annotating Resource Class   (These are class level)
@Path:
When we create a root resource, we must provide the @Path annotation.The annotation contains the URI path for the resource . Eg., /cart .If we create a sub-resource, the @Path is not required.


@RestResource
The @RestResource must be present for all resource classes, as it specifies the ID of the resource. The ID is a string that uniquely identifies the resource and does not change when the context and/or version change.



Annotating Endpoint Methods
@Endpoint 
We can use the @Endpoint annotation to annotate resource methods and configure the various setting for the endpoint, such as transactions, security, and cache.

 We can use the following annotations when creating an endpoint method:

@Path 


 When creating an endpoint method, use the @Path annotation if there are additional path elements beyond the resource path or the sub-resource methods. The annotation is not required if the path is empty.


@GET, @POST, @PUT, @PATCH, and, @DELETE


These annotations are used as necessary when working with the endpoint. But we used @POST annotation most of the times.


@Endpoint

This annotation identifies the endpoint with a unique string ID. The ID does not change if the context or version changes. If the endpoint is for a singular resource, the isSingular attribute must be true.


Naming Endpoint Annotations

When you define an ID in a @Endpoint annotation, it is best to name it as a combination of the resource URI and the method request type. For example, a product resource might be:


@Endpoint(id="/products/{productId}#GET, isSingular = true)

Ordering Endpoint Annotations
When creating an endpoint, it is best to be consistent with the order of the annotations. For example:

@GET
@Path("/{productId}")

@Endpoint(id="/products/{productId}#GET", isSingular = true)

The endpoint method should be public.
If the endpoint response is a representation of the resource, the return type should be RepresentationModel. The model holds the data needed to support hypermedia formats. It includes properties, links, the representation state and any embedded resources URI that should be included in the response.
Note: In some cases, the endpoint method may return only the status code. If this is the case, you should use the javax.ws.rs.core.Response class that has the javax.ws.rs.core.Response.Status enum to set and return the proper status code.


RepresentationModel
An endpoint should return an instance of a RepresentationModel if the response is a representation of the resource. You can obtain an instance of a RepresentationModel by using the build method of a RepresentationModel.Builder instance. The RepresentationModel.Builder method begins a RepresentationModel.Builder instance. The following builder methods are available.

1) State Builder Method
2)Properties Builder Methods
3)Links Builder Method
4)Self Builder Methods

1) State Builder Method

The state(Object pState) bean or dynamic bean holds the current state of the resource, for example a repository item or an Order bean. This object is filtered and rendered by the framework into the media type, and can be null, although it would normally be null only for collection resources.


Sample code for  State Builder Method.We used this as part of our implementation.
xxxPayLoad guestPayLoad=new xxxPayLoad();
RepresentationModel.Builder builder = null;
builder = new RepresentationModel.Builder().state(xxxPayLoad);
return builder.build(); (return type is RepresentationModel)

 Associating a Validator with an Endpoint

@Endpoint(id = "/addGroup#POST", filterId = "xx.abc-Filter",validatorId="xxValidator", isSingular = true)


Sample code @payloadSchema.xml

<schema include-all-simple="false" id="xxxValidator"
treat-container-as-bean="true">
<bean class="org.json.JSONObject" />
<property name="cxxl" writable="false" required="true" />
<property name="bxxd" writable="false" required="true" />
<property name="exxd" writable="false" required="true" />
<property name="gxxxe" writable="false" required="true" />
</schema>

 Associating a Filter with an Endpoint
Using the filterId property of the @Endpoint annotation, which is described in Associating a Filter with an Endpoint

@Endpoint(id = "/addGroup#POST", filterId = "xx.abc-Filter",validatorId="xxValidator", isSingular = true)

Sample code @PayloadSchema.xml

<schema include-all-simple="false" id="xx.abc-Filter"
treat-container-as-bean="true">
<bean
class="com.xxabc.response.xxxPayload" />
<bean-property use-schema-id="Status-VO" name="status" writable="false" />
<bean-property use-schema-id="ItemInfo" name="itemInfo" writable="false" />
</schema>



PayloadSchemaRegistry
Need to configure/add the cusatom payload schemas.
atg/dynamo/service/payloadschema/PayloadSchemaRegistry.properties
definitionFile=atg/dynamo/service/payloadschema/payloadSchema.xml

Customizing Property Output

We can customize the output of properties by using the property-customizer attribute of the property element.
This attribute specifies a Nucleus component that is a class implementing the atg.service.filter.bean.PropertyCustomizer interface.

Sample payloadSchema code

<schema id="orders.id-Full" include-all-simple="true">
  <bean class="atg.commerce.order.Order"/>
  <property name="orderTotal" property-customizer="atg.PricePropertyCustomizer"/>
</schema>

The following example shows a PricePropertyCustomizer:

public class PricePropertyCustomizer implements PropertyCustomizer {
  public Object getPropertyValue(Object pTargetObject, String pPropertyName,
      Map<String,String> pAttributes)
          throws BeanFilterException {
    Double value = (Double)DynamicBeans.getPropertyValue(pTargetObject,
        pPropertyName);
    return "£" + String.valueOf(value);
  }
}

embedded element
We can use the embedded element of the payload schema XML for filtering embedded resources. By default, embedded resources and sub-resources occur in the response as a self link

<schema id="orders.id-Full" include-all-simple="true">
  <bean class="atg.TestOrder"/>
  <embedded name="commerceItems" hidden="false"/>
</schema>

{
  "id": "o12345",
  "state": "SUBMITTED",
  ...
  "commerceItems": [
    {
      "id": "ci1",
      "quantity": "3",
      "productId": "prod001"
      "_links":{
        "self":{
          "href": "test/orders/commerceitems/ci1"
        }
      }
    },
    {
      "id": "ci2",
      "quantity": "2",
      "productId": "prod006"
      "_links":{
        "self":{
          "href": "test/orders/commerceitems/ci2"
        }
      }
    },
  ],

}

The embedded element has a use-schema-id attribute that We can use to override the schema defined by an endpoint when the embedded resource or sub-resource is provided as part of the root response.

<schema id="orders.id-Summary">
    <bean class="java.util.List"/>
    <embedded name="items" expand="true" use-schema-id="
      commercieitem.id-Summary"/>
</schema>

Even we can use as part of the validator and filter.

Collections and embedded 
<schema id="products-Summary">
  <bean class="java.util.Collection"/>
  <embedded name="items" hidden="false" use-schema-id="products.id-Summary"/>
</schema>

Using Query Parameters

We can filter the response output using query parameters.

 There are many . 

NOTE: Web service guide

https://docs.oracle.com/cd/E69533_01/Platform.11-3/ATGWSFrameGuide/html/s0604usingthepayloadschemaregistry01.html



The OOTB table is getting incremented  for every service request by RAXRS.

(/atg/dynamo/service/jaxrs/ETagService/)

Because of this, DB connectivity issue occurs and crashes. So to avoid this issue, disabled the the above service (ETagService).

ATG BCC Targeters info

It is explained at oracle ATGPersProgGuide.pdf

atg/targeting/TargetingSourceMap.properties

sourceMap+=xxComponent=com/xx/yy/xxComponent

----


 This section covers the targeters bulk import steps.These steps are executed in one of my project Once We did initial BCC setup via CIM.

1)Need to create one folder "ImportTargeter" in the path /opt/ATG/ATG11.1/home.
2)And copy the required targeters in this folder.
3)Need to prepare any manifest file(targeter.MF) as mentioned below.
4)Need to set the DAS/lib/classes.jar and RL/lib/RLClient.jar to the Classpath.
4)Before executing the RLClient.bat/RLClient.sh file, need to set to change in RLClient.bat file as at the end.
  java atg.repository.loader.RLClient %1 %2 %3 %4 %5 %6 %7 %8 %9

 If we have change like this, the DAS/lib/classes.jar and RL/lib/RLClient.jar  can be used.Otherwise, DAS/lib/classes.jar won't be available and got the error as one of the class is not available from classes.jar

 5)Change below configuration at BCC dynamo administration.
     dyn/admin/nucleus/atg/epub/file/VersionedLoaderEventListener/

rootPath /opt/ATG/ATG11.1/home/ImportTargeter
createProcesses TRUE

6)Execute the following command.
./RLClient.sh -m /opt/ATG/ATG11.1/ImportTargeter/targeter.mf -h 150.21.21.162 -r 8861

Where : 8861 is the RMI port of Publishing server and  150.21.21.162 is publishing server host number


7) Environment Variable Configuration at Linux box.

Set the CLASSPATH for below Jars. ${DYNAMO_ROOT}/RL/lib/RLClient.jar ${DYNAMO_ROOT}/DAS/lib/classes.jar

For Linux: Go to the below path set the ClassPath values into the .bash_profile file. /home/admin/.bash_profile

Once the Classpath is set at Linubox, please close all Linux box screens. Then only Classpath entry will work.


The following file should be created.For each path , there should be one <add  script.
i.e., Homepage --> one add tag
i.,.,Footer -->one add tag.

If  these folders are available  already at  BCC, no add to <add tag for each folder.

targeter.MF

<manifest>
<add type-mapping="/atg/epub/file/typemappers/FileFolderTypeMapping">
/opt/ATG/ATG11.1/home/ImportTargeter/atg/registry/RepositoryTargeters/Homepage
</add>
<add type-mapping="/atg/epub/file/typemappers/FileFolderTypeMapping">
/opt/ATG/ATG11.1/home/ImportTargeter/atg/registry/RepositoryTargeters/Homepage/Footer
</add>
<add type-mapping="/atg/epub/file/typemappers/TargeterTypeMapping"> /opt/ATG/ATG11.1/home/ImportTargeter/atg/registry/RepositoryTargeters/Homepage/Footer/CookieMessagePolicyTargeter.properties
</add>
</manifest>

Attached the Import targeter zip file which contains all the Targeters and Manifest file.This need to  be copied as mentioned in the above steps and execute the RLClient.sh as mentioned above.