Workflow Progress Table

This topic explains each step in the Workflow Progress section of the Load page and how to execute each step.

The Migration Workflow is the process of migrating instance data into CDMS, not the Study Design itself. It involves preparatory steps such as uploading source data and creating a Load and concludes with Reconciling Listings.

Understanding the actions that take place in each step of the workflow is useful for troubleshooting Migration errors.

Learn how to create a load and then start the load.

After you create a load and select your mapping configurations, the following steps appear in a table in the Workflow Progress section:

1. Validate & Stage
2. Deactivating Rules
3. Run
4. Med Coding
5. Form Submission
6. Post-Run
7. Dynamic Rules
8. Subject Statuses
9. Reactivate Rules
10. Reconcile Listings

All workflow steps are required except Dynamic Rules and Subject Statuses. You can execute steps individually or all at once using the Execute All Steps feature. You’ll receive an in-app and email notification when each step is complete. The Status column also identifies the progress of each step. The following statuses are provided in the table:

• Not Started
• Queued
• In Progress
• Success
• Failed
• Skip/Skipped
• Stopping

Validate & Stage

The first step, Validate & Stage, is initiated at the start of the load. Pre-processing takes place during this step, including data validation and staging. It’s comprised of four phases:

• Parse
• Validation
• Add Study ID
• Stage

During the parsing phase, the system reads the YAML and CSV files, applies changes to the CSV file from the YAML file, and then parses the CSV file. During the Validation phase, the Study Design is obtained from CDMS and each record is validated, including date formats and form statuses. The Migration Vault to CDMS connection is automated and does not require user engagement.

If an error occurs, the Validation phase fails, and the user is notified by email. Details regarding the failure are provided in the log file associated with the step. During the Add Study ID phase, definition IDs are attached to records and those records are then sent to staging during the Staging phase.

1. Click Execute or Execute > Execute All Steps to Validate & Stage.
2. Click Validate Only to validate and stage separately.
3. Click Stage Load to stage the load when you’re ready.
4. Click Log File to download a CSV of log file errors and warnings.

Deactivating Rules

This step enables derivation rules and deactivates all other rules. All rules deactivated during this step are reactivated in Step 9: Reactivate Rules. Derivation rules are executed in Step 3: Run.

1. Click Execute > Execute All Steps to deactivate rules.
2. Click Log File to download a CSV of the deactivated rules.

Run

During the Run step, data is migrated from staging to CDMS Vault and derivation rules are executed.

1. Click Execute > Execute All Steps to migrate the data from the staging database into EDC.
2. Click View Details to view step results on the Run Summary page.
3. Click Log File to download a CSV of log file errors and warnings.

Med Coding

During Med Coding, coding data is migrated from staging to CDMS Vault. Coding data must be prepared prior to migration.

Learn about Preparing Coding Data for Migration.

1. Click Execute > Execute All Steps to migrate coding data.
2. Click Continue in the confirmation dialog.
3. Click View Details to view the Med Coding Summary page.

Form Submission

This step allows you to submit forms, and when appropriate, run derivation rules upon submission. During this step, attributes are not applied, ensuring greater consistency across loads and preventing concurrent asynchronous jobs.

1. Click Execute > Execute All Steps to submit forms.
2. Click Continue in the confirmation dialog.
3. Click View Details to view results in the Form Submission table.

Post-Run

The Post-Run step is responsible for the following actions:

• Applying attributes
• Creating queries
• Executing other post-processes

1. Click Execute > Execute All Steps to initiate post-run processes.
2. Click Continue in the confirmation dialog.
3. Click View Details to view the Post-Run Summary page.
4. Click Log File to download a CSV of log file errors and warnings.

Dynamic Rules

This step executes or skips the Dynamic Rules. If it fails and rules are reset, the workflow progress table updates to reflect a Success status for the Reactivate Rules step.

1. Click Execute > Execute All Steps to activate and run Dynamic Rules.
2. Click Skip to skip this step.
3. If executing, click Continue in the confirmation dialog.
4. Click Log File to download a CSV of the dynamic rules that were executed.

Subject Statuses

With this step, Subject Status rules are either executed or skipped in the Workflow Progress table.

When executed, the system automatically sets subject statuses in the following order:

1. Pre Screen

2. Consented

3. In Screening

4. Screen Failure

5. Enrolled

6. Randomized

7. Started Treatment

8. End of Treatment

9. Withdrawn

10. Started Follow Up

11. Lost to Follow Up

12. Complete

1. Click Execute > Execute All Steps to activate and run Subject Status rules.
2. Click Skip to skip this step.
3. If executing, click Continue in the confirmation dialog.
4. Click View Details to view results and Log File to download a CSV of log file errors.

If Execute All Steps is selected in any step prior, the system attempts to execute the entire workflow. In this case, the Subject Statuses step will only work if the previous step was successful. If the previous step failed, the step will not auto-start. Once the previous step is confirmed with a status of Success, the Subject Status rules deactivated in step two are reactivated and executed.

Reactivate Rules

This step reactivates every rule previously deactivated as part of Step 2: Deactivate Rules.

1. Click Execute > Execute All Steps to Reactivate Rules.
2. Click Continue in the confirmation dialog.
3. Click Log File to download a CSV of the reactivated rules.

The Reset Rules after Failure option performs the same actions as the Reactivate Rules step. This option appears next to any step that fails between steps three and eight. If clicked, all the rules deactivated as part of step two are reactivated.

Reconcile Listings

This step is a job that can be performed after a Load is complete. It synchronizes the CDMS data with the EDC UI to ensure that the migration was a success. This job can only be performed by users with System Administrator and Vault Owner security profiles. There is no Skip option for this step, allowing users to return at any time to execute the job if needed.

1. Click Execute to run the job.
2. Click Continue in the confirmation dialog.

Learn more about Log Files.