Workflow Progress Table

This topic explains each step in the Workflow Progress section of the Load page and how to execute each step.

The Migration Workflow is the process of migrating instance data into EDC, not the Study Design itself. It involves preparatory steps such as uploading source data and creating a Load and concludes with Reconciling Listings.

Understanding the actions that take place in each step of the workflow is useful for troubleshooting Migration errors.

Learn how to create a load and then start the load.

After you create a load and select your mapping configurations, the following steps appear in a table in the Workflow Progress section:

1. Validate & Stage
2. Deactivating Rules
3. Run
4. Med Coding
5. Form Submission
6. Post-Run
7. Dynamic Rules
8. Subject Statuses
9. Reactivate Rules
10. Reconcile Listings

All workflow steps are required except Dynamic Rules and Subject Statuses. You can execute steps individually or all at once using the Execute All Steps feature. You’ll receive an in-app and email notification when each step is complete. The Status column also identifies the progress of each step. The following statuses are provided in the table:

• Not Started
• Queued
• In Progress
• Success
• Failed
• Skip/Skipped
• Stopping

Validate & Stage

The first step, Validate & Stage, is initiated at the start of the load. Pre-processing takes place during this step, including data validation and staging. It’s comprised of four phases:

• Parse
• Validation
• Add Study ID
• Stage

During the parsing phase, the system reads the YAML and CSV files, applies changes to the CSV file from the YAML file, and then parses the CSV file. During the Validation phase, the Study Design is obtained from CDMS and each record is validated, including date formats and form statuses. The Migration Vault to CDMS connection is automated and does not require user engagement.

If an error occurs, the Validation phase fails, and the user is notified by email. Details regarding the failure are provided in the log file associated with the step. During the Add Study ID phase, definition IDs are attached to records and those records are then sent to staging during the Staging phase.

1. Click Execute or Execute > Execute All Steps to Validate & Stage.
2. Click Validate Only to validate and stage separately.
3. Click Stage Load to stage the load when you’re ready.
4. Optional: Click Log File to download a CSV of log file errors and warnings.

Deactivating Rules

This step enables derivation rules and deactivates all other rules. All rules deactivated during this step are reactivated in Step 9: Reactivate Rules. Derivation rules are executed in Step 3: Run.

1. Click Execute > Execute All Steps to deactivate rules.
2. Optional: Click Log File to download a CSV of the deactivated rules.

Run

During the Run step, data is migrated from staging to CDMS Vault and derivation rules are executed.

1. Click Execute > Execute All Steps to migrate the data from the staging database into EDC.
2. Optional: Click View Details to view step results on the Run Summary page.
3. Optional: Click Log File to download a CSV of log file errors and warnings.

Med Coding

During Med Coding, coding data is migrated from staging to CDMS Vault. Coding data must be prepared prior to migration.

To migrate coding data:

1. Click Execute > Execute Step to migrate coding data.
2. Click Continue in the confirmation dialog.
3. Optional: Click View Details to navigate to the Summary page and view the Med Coding Summary table.

Med Coding Summary Table

The Med Coding Summary table highlights the progress of the Med Coding step: Create and Send. The Create stage references the point at which coding requests are created and contains the counts for that stage. The Send stage references the point at which coding requests are sent to EDC Coder and contains the counts for those requests. The counts for the Send stage are only populated when the Create stage is fully complete.

Each stage is given a status of Pending, Success, or Failed. When the Med Coding step succeeds, the Create and *Send *counts match. Differences in the count indicate a failure.

For example: If 50 records are successfully created and two (2) records fail, the Send stage shows 50 as Success and two (2) as Pending. The Pending count reflects the number of records that failed the Create step. If a record fails during the Send stage, it appears under the Failed column for Send.

All rows are exported into the Migration Report.

Form Submission

This step allows you to submit forms, and when appropriate, run derivation rules upon submission. During this step, attributes are not applied, ensuring greater consistency across loads and preventing concurrent asynchronous jobs.

1. Click Execute > Execute All Steps to submit forms.
2. Click Continue in the confirmation dialog.
3. Optional: Click View Details to view results in the Form Submission table.

Post-Run

The Post-Run step is responsible for the following actions:

• Applying attributes
• Creating queries
• Executing other post-processes

1. Click Execute > Execute All Steps to initiate post-run processes.
2. Click Continue in the confirmation dialog.
3. Optional: Click View Details to view the Post-Run Summary page.
4. Optional: Click Log File to download a CSV of log file errors and warnings.

Dynamic Rules

This step executes or skips the Dynamic Rules. If it fails and rules are reset, the workflow progress table updates to reflect a Success status for the Reactivate Rules step.

1. Click Execute > Execute All Steps to activate and run Dynamic Rules.
2. Click Skip to skip this step.
3. If executing, click Continue in the confirmation dialog.
4. Optional: Click Log File to download a CSV of the dynamic rules that were executed.

Subject Statuses

With this step, Subject Status rules are either executed or skipped in the Workflow Progress table.

When executed, the system automatically sets subject statuses in the following order:

1. Pre Screen

2. Consented

3. In Screening

4. Screen Failure

5. Enrolled

6. Randomized

7. Started Treatment

8. End of Treatment

9. Withdrawn

10. Started Follow Up

11. Lost to Follow Up

12. Complete

1. Click Execute > Execute All Steps to activate and run Subject Status rules.
2. Click Skip to skip this step.
3. If executing, click Continue in the confirmation dialog.
4. Optional: Click View Details to view results and Log File to download a CSV of log file errors.

If Execute All Steps is selected in any step prior, the system attempts to execute the entire workflow. In this case, the Subject Statuses step will only work if the previous step was successful. If the previous step failed, the step will not auto-start. Once the previous step is confirmed with a status of Success, the Subject Status rules deactivated in step two are reactivated and executed.

Reactivate Rules

This step reactivates every rule previously deactivated as part of Step 2: Deactivate Rules.

1. Click Execute > Execute All Steps to Reactivate Rules.
2. Click Continue in the confirmation dialog.
3. Optional: Click Log File to download a CSV of the reactivated rules.

The Reset Rules after Failure option performs the same actions as the Reactivate Rules step. This option appears next to any step that fails between steps three and eight. If clicked, all the rules deactivated as part of step two are reactivated.

Reconcile Listings

This step is a job that can be performed after a Load is complete. It synchronizes the CDMS data with the EDC UI to ensure that the migration was a success. This job can only be performed by users with System Administrator and Vault Owner security profiles. There is no Skip option for this step, allowing users to return at any time to execute the job if needed.

1. Click Execute to run the job.
2. Click Continue in the confirmation dialog.

Learn more about Log Files.