Creating and editing integration jobs

Setup Information (First Section)

Create a descriptive name to identify the Job, such as Rebuild Search Index.

Refresh: This option will cause the job to pull data from the integrated system and transcribe it into the B2B Commerce database

Submit: This option will cause the job to take a dataset as a result of either a Preprocessor or a dataset submitted via code and make it available to the Windows Integration Service (WIS). This type of job is typically used for Customer creation, order creation, and payment submission

Report: This option will cause the job to run a data query, typically locally, to pull ecommerce data out and make the data available for a report

Execute: This option will cause the job to run a SQL statement or stored procedure. Typically used for scheduling maintenance and archiving tasks

This drop-down is used to select the connection that will be used for the job. The drop-down will be populated by the names of the connections defined in the Integration Connection screen.

The connection named Internal is defined by the system and is used for Imports and Exports and for any jobs that should be initiated directly by the web server instead of the WIS.

This drop-down is used to select the plug-in for pre-processing the job. None is an acceptable option and would always be appropriate for Execution jobs.

The preprocess is a plug-in that runs before the job is submitted to the WIS/Internal WIS.

This is the processor plug-in running on WIS that will be sent down with the task DTO packet to indicate to WIS what process to run. None would be an option as well, especially for Execution jobs.

This is the processor that runs once the result is returned.

Only processors that support the job type listed will be available in the drop-down. This is the plug-in that has the largest list of function-specific options.

If a stored procedure is desired to be run, then the Postprocessor should be set to ExecuteStoredProcedure and a job parameter called StoredProcedureName should be set up with the specific procedure to run.

None is an available option.

Provide a detailed description of what purpose the Job serves and any other information that would be helpful to another user who may use or inherit the job in the future.

Toggle turns on the delta dataset function to do a compare of the acquired data to the last saved "good" set (if any) and construct an add/change/delete dataset. This function should only be used for refresh job types.

This option should only be used if the integration process is taking a full snapshot of the source data. The WIS essentially takes this snapshot and compares it against the last saved snapshot and sends up the differences. If you are processing flat files, for example, it would not be an appropriate selection and could end up disabling/deleting records that are not intended to be modified.

If the Debugging Enabled is for the associated Connection, then all jobs running under that Connection will have debug messages. If it is not on, then debugging will only occur for those Job Definitions where it is specifically turned on.

Debugging should only be enabled when troubleshooting an issue. This is true for both the Integration Job itself, as well as Connections. Enabling Debugging causes jobs to run longer and collect far more additional information for every job tied to it.

Notifications Section

This field is used to identify the email addresses to send a notification email to (if applicable). If multiple email addresses are used, they should be comma delimited.

Completion: This condition is used to send an email once the selected integration job has completed, regardless of success or failure

Success: This condition is used to send an email once the selected integration job has successfully completed

Failure: This condition is used to send an email once the selected integration job has completed in a status other than Success.

This drop-down is used to select the email template that will be used for the notification email if an email is being generated. If not template is selected, the system will use a default notification template.

This drop-down is used to select another job to immediately spawn and run after the currently selected job has completed, depending on the link condition.

SuccessOnly: Linked job will only run if the current job was successful.

SuccessOrWarning: Linked job will run if the current job was successful or if the job triggered a warning.

SuccessWarningErrorOrFailure: Linked job will run if the current job was successful, or if the job triggered a warning, error or even if the job failed.

SuccessWarningOrError: Linked job will run if the current job was successful or if the job triggered a warning or error.

Run Options Section

This field will create a threshold for error log messages (you can check them in the History tab). If this threshold is passed, the job execution stops, already completed changes try to commit to database and we get "The job was failed because the maximum amount of allowed errors/warning ( {logCount} ) has been exceeded." log on history tab. If 0 is chosen for one of this setting, it will not limit number of errors encountered and will run to completion.

Entering a value of 0 = unlimited.

This field will create a threshold for warning log messages (you can check them in the History tab). If this threshold is passed, the job execution stops, already completed changes try to commit to database and we get "The job was failed because the maximum amount of allowed errors/warning ( {logCount} ) has been exceeded." log on history tab. If 0 is chosen for one of this setting, it will not limit number of warnings encountered and will run to completion.

Entering a value of 0 = unlimited.

This setting will skip delete execution if the number of records expected to be deleted/deactivated exceeds the specified percentage. The intention is to disallow accidental deletion of masses of records.

If the job has multiple steps, only the delete action on the current step will be skipped - it will still execute subsequent steps.

If 0 is chosen for this setting, it will use 20% as a default.

This setting allows each job to have a custom batch size or to set a different default value for batch sizes. When the job runs, if there is a batch size > 0 in the job definition, the job will use this value instead of the default setting value.

Default processing batch size is 50.

There is a hard limit on the number of parameters you can pass into a parameterized SQL statement of 2,100, which is what the batching is doing. The performance will NOT necessarily be better as the number of records goes up, as it is highly dependent on the number of values that make up the natural key and what other related tables you are accessing for lookups and child collections. 

If you target a table like Specifications, where the natural key = ProductID, CategoryId and Name, each record contains three parameters. The maximum size of the batch in this case would be 700 (2,100/3=700). 

Optimizely recommends that you test with different values until you see either reduced overall performance or run out of a parameter limit. As a rule of thumb, only jobs that take longer than five minutes benefit significantly from tuning the batch size. Optimizely does not recommend going above  a batch size of 500, because if there is a problem with a given record in the batch, the system reverts to posting records one at a time, which, if the batch size is too large, could eliminate most or all of the gain of batching in the first place.

Recurrence Section

This toggle is used to flag the job as a recurring job.

If this flag is turned on, when the record is saved, the system will automatically create a queued job at the next appropriate time that matches the start date/time and repeat cycle. The Job will need to be run manually first before the change is picked up. 

This field is used to select the starting date and time for the recurring job to operate.

This field is used to select the ending date and time for the recurring job to operate. It can be left blank which indicates that the job has no end date.

Setup Information (First Section)

This field represents the execution sequence that will be used to run the steps - the user should make sure to enter a value other than 0.

This field is an informational field to identify what is being done

This field defines the B2B Commerce target object for Refresh jobs.

This is a drop-down of the connections available. It may optionally be overridden from the one at the job definition level.

If the specific step needs a custom processor, this field may be set to instruct the WIS to use a different set of code to process this step with. This should only ever be necessary on multi-step jobs.

The Action is applicable only for Refresh jobs and helps define what action to take when a record is not including in a full dataset or a delete record is sent up from a delta dataset.

Delete Record: This would attempt to actually delete the record - this could potentially cause referential integrity issues or fail due to a database constraint

Ignore: Do not do anything

Set Field: This option will expose a Field To Set control to select the field to set. This is used typically to "deactivate" an item by setting its "Active" boolean value or DeactivateOn date but could be used to set most fields to a specified value

This field is only available for refresh jobs where Set Field is chosen for the Action in the previous field.

This field lists the fields in the object that can be set.

During the transcription/post processor, the system would set the field depending on the field type - if it is a date field, we would set it as the current date, otherwise set it to the value as specified in the Field Value

The user must take care to set the Value To Set as a valid value for the field being set. For example, if the field is a boolean value, use 0 or 1 (or true/false) as the selected values. A blank date will set a date field to the then current date/time as it is being set.

The integration connection will define the directory to scan for files and this field is designed to select a file mask for files that match.

For example, if the files have a leading name such as "ProductFeed_xxxx" where xxxx is a date/time stamp, use ProductFeed*.* as the mask.

If several file names are valid for the step, you can enter multiple masks using a comma to separate them. You can also signify a subdirectory as part of the mask so that the system will search for the files within a subdirectory, thus avoiding having to set up separate connections.

This field defines how you wish the system to behave under different conditions. This value should be set based on when the integration job runs and if you expect any files to be present or not and how you wish to be notified if there are no files that match the mask entered.

Error: if the file is missing, it will issue an Error message in the log and increment the #/errors encountered. The job will ultimately fail with an error if no file is present.

Warning: if the file is missing, it will issue a Warning message in the log and increment the #/warnings encountered. The job will ultimately fail with warnings if no errors are encountered after the missing file condition.

Ignore: this indicates that no files present is a normal and expected condition and an Info message will be generated and the job would end as successful as long as there are no other steps with other issues.

This should be turned on if the flat file has a header in it showing the column names - this setting will skip the first row. If this setting is not on, all rows will be processed.

For Refresh jobs, this is essentially the first part of a "SELECT" clause and defines the columns to be retrieved from the remote data source. For flat files, this is a simple list of columns.

The #/columns defined must match the data in the flat file if a flat file is being used or an error condition will occur. You do not need to map all the columns, but you do need to define them all for ingestion.

You can use whatever functions are supported by the connection. For example, if the source db is SQL Server, you can use a SUBSTRING command or even a subquery.

For Refresh jobs using a database connection, this indicates the tables to use and the join conditions

For Refresh jobs, this is the WHERE clause of a SELECT statement and defines any data-based conditional expressions. Use single quotes for static variables such as Status = "A". This can also be used to conditionally include records from a flat file

You can also use a parameterized value from the Application Settings by setting the name of the setting in braces such as

transdt > getdate() - {ERP_Lookbackdays}

This is a special-purpose field to define an override where clause when the job is set up with a step parameter defined and was intentionally created to allow for the same job to be used to refresh all customers or a single customer.

The intent is to have a separate WHERE clause that is invoked only if there are parameters on the job. If there are any parameters defined on the job the parameterized where clause will be used.

To use the parameter within the WHERE clause, prefix the name of the parameter with an @.

In 4.2 we have eliminated this capability and, therefore this field may be eliminated in the future.

When an SAP DotNetConnector option is used, several different fields will appear on the details tab.

This represents the specific BAPI within SAP to call

BAPIs can return Function (property list), Structure (like a single record with properties) or an actual Table.

Entering the table name will refer to the table itself that is returned.

Entering the function name surrounded by {curly braces} will denote that the function data is being returned.

Entering the structure name surrounded by [square brackets] will denote the structure being returned.

Note that the Bapi Info can define the specific values from any of these by using the same convention to reference data in any of the 3. If something is entered here, it does not need to be referenced again in the list of properties - it becomes the default.

Parameters can be passed into the DotNetConnector either as individual parameters as defined in the Step Parameters or they can be grouped together into a Parameter Table.

Enter the parameter table name if the parameters should be consolidated and grouped together as a parameter table.

This drop-down indicates in context the type of data we are either reading (refresh) or writing (submit)

Field: Indicates a data field from the object (submit) or ERP source (refresh).

Static Value: A specifically entered value - used when you want the data value to be a fixed value.

Application Setting: The value comes from the application setting - it becomes a static value from the application setting.

If this is selected, the From Property will be replaced with a list of valid application settings.

Content: This is a special field type and is reserved for refresh jobs. Since content has a context defined by language, persona and device, the job can define either the default values, static values or values from the import data.

Lookup: This is used for looking up an object to be used in a parent/child relationship. This is a very important field type and is used when referencing to another entity in the system. Only entities of the target object should be display.

An example is if we are refreshing Products, then a valid Lookup would be Vendor. The "natural key"� is used to look up the alternate entity (that is Vendor Number) and then embed the underlying Id field into the Product.

For example, if we are refreshing Inventory (ProductWarehouse entity), we would do a lookup on the Product using the Product Number and a lookup on the Warehouse using the Warehouse Name.

If Lookup is selected, the user must first select the To Property after which the From Property will be shown

Child Collection: Used for cross matrix selections - assignment records that have no data themselves. Notice that we used ProductWarehouse using Lookup mapping because there is additional data we want to capture for that record.

The Child Collection is used when there is an assignment but no actual data added onto that assignment. An example is WebSiteCurrency - since there is no specific entity/target object, we import using Website as the target and the child collection of Currency. The data stream must have the natural key of the website (name) and of the currency (currency code) to create the record.

As a general approach, when integrating with a child collection, go to the collection with the fewest children for performance. For example, do not update Category with a list of Products since Categories have lots of products. Instead, update Product with a list of categories, because each product is in far fewer categories. This approach tends to solve performance issues.

This defines the source of the data that will be used.

For a refresh job, it will give you a list of the entries from the SELECT clause. It may not always display properly if there are functions and subselects in the query, so the user may type in the actual value intended to use.

This defines the target of the data that will be transcribed.

For a refresh job, it will give you a list of the entries in the Target Object.

This is reserved for Content field types and will provide a list of the data elements defined in the query and a list of available language codes in the system. Leaving this field empty will use the System Default.

This is reserved for Content field types and will provide a list of the data elements defined in the query and a list of available personas in the system. Leaving this field empty will use the System Default.

This is reserved for Content field types and will provide a list of the data elements defined in the query and a list of available device types as defined in the System List. Leaving this field empty will use the System Default.

This is only used for refresh jobs and lookup field types.

If the related entity cannot be found, the system can determine how the job should log it - either as an error, a warning or to ignore it altogether.

If this flag is set to true, the data will always be overwritten from the data feed on a refresh.

This flag is intended to allow the refresh to establish an initial value but then never overwrite that value since the user would be responsible for maintaining it going forward. A good example is the Product Title field - we want to populate it initially from the ERP but then let the site administrators change that value and no worry about having it overwritten.

There is no need to set this for key fields - anything that is a dataset key should NOT be set to overwrite.

There is no defined behavior for this field on submit jobs.

This field is used to identify the order in which the parameters are either prompted for or passed into the processors. This field is hidden by default and is typically not necessary to be filled in.

This field define the name of the parameter - it should be unique within the job definition

This drop-down is used to select the type of value for the parameter

The default for the entry.

If the value type is Application Setting, then the value set here represents the actual name of the setting. The value of that setting will be used at runtime.

If the value type is RelativeDateTime, then the value is a simple integer indicating the number of days forward or backwards from the current date. This is useful if you want to run an integration that "looks back"� a set number of days.