Workflow Progress Table

This topic explains each step in the Workflow Progress section of the Load page and how to execute each step.

The migration workflow migrates instance data into EDC rather than the study design itself. It begins with preparatory steps, such as uploading source data and creating a load, and concludes with the Reactivate Rules workflow step. Understanding each step in the workflow helps you troubleshoot migration errors.

After you create a load and select your mapping configurations, the Workflow Progress table displays the following steps:

1. Validate & Stage
2. Deactivating Rules
3. Run
4. Med Coding
5. Form Submission
6. Post-Run
7. Reconcile Listings
8. Dynamic Rules
9. Subject Statuses
10. Freeze & Lock
11. Reconcile Freeze & Lock
12. Reactivate Rules

All workflow steps are required except Dynamic Rules and Subject Statuses. You can execute steps individually or all at once using the Execute All Steps feature. You’ll receive an in-app and email notification when each step is complete. The Status column also identifies the progress of each step. The following statuses are provided in the table:

• Not Started
• Queued
• In Progress
• Success
• Failed
• Skip/Skipped
• Stopping

Validate & Stage

The Validate & Stage step is the first step in the migration process. It verifies that source data matches the target study’s structure before the system begins migrating.

Validate & Stage comprises four phases:

• Parsing
• Validation
• Add Study ID
• Staging

During the Parsing phase, the system reads the YAML and CSV files, applies changes to the CSV file from the YAML file, and then parses the CSV file.

During the Validation phase, the system obtains the Study Design from EDC and validates each record, including date formats, form statuses, and sequence numbers for event groups, forms, and item groups.

The Migrator logs an error in the log file and stops the migration process if a sequence number does any of the following:

• Contains letters or text
• Includes decimal places other than .0 (for example, the Migrator will accept 1.0 but reject 1.2)
• Falls below the minimum starting threshold set in the header or logheader YAML files (for example, source data records contain sequence numbers that begin with 0, but formSequenceStartingNumber in the header file is set to 1)

During the Add Study ID phase, the system attaches study definition IDs to records and sends those records to staging during the Staging phase.

To initiate the validation process:

1. Navigate to your Load.
2. Click Workflow Progress to expand that section.
3. Hover over the Validate & Stage step to show the Execute menu.
4. Select Execute to execute both Validation and Staging.
5. Click Validate Only to validate without staging, then click Stage Load to stage the load when you’re ready.
6. Optional: Click View Details to open the Validate & Stage Results page.
7. Optional: Click Log File to download a CSV attachment of errors and warnings.

Error Handling & Logging

If the Validation or Staging phases fail, the system captures all errors and warnings in a single log file (for example, values that exceed the character limits). The system also notifies the migrator of the failure via email, which contains a link to view the Validate & Stage Results page.

You can review errors and warnings in two ways: via a summary in the Validate & Stage Results > Status Detail column or by downloading the complete log file. The Status Detail column displays a summary (up to 32,000 characters) of errors and warnings per YAML file. Click Show more in the Status Detail column to expand the summary in a new window. Click Close to close the window.

Deactivating Rules

This step enables derivation rules and deactivates all other rules. The Reactivate Rules step then reactivates them. Derivation rules are executed in step 3, Run.

1. Click Execute > Execute All Steps to deactivate rules.
2. Optional: Click Log File to download a CSV of the deactivated rules.

Run

During the Run step, data is migrated from staging to EDC and derivation rules are executed.

1. Click Execute > Execute All Steps to migrate the data from the staging database into EDC.
2. Optional: Click View Details to view step results on the Run Summary page.
3. Optional: Click Log File to download a CSV of log file errors and warnings.

Med Coding

During Med Coding, coding data is migrated from staging to CDMS Vault. Coding data must be prepared prior to migration.

To migrate coding data:

1. Click Execute > Execute Step to migrate coding data.
2. Click Continue in the confirmation dialog.
3. Optional: Click View Details to navigate to the Summary page and view the Med Coding Summary table.

Med Coding Summary Table

The Med Coding Summary table highlights the progress of the Med Coding step: Create and Send. The Create stage references the point at which coding requests are created and contains the counts for that stage. The Send stage references the point at which coding requests are sent to EDC Coder and contains the counts for those requests. The counts for the Send stage are only populated when the Create stage is fully complete.

Each stage is given a status of Pending, Success, or Failed. When the Med Coding step succeeds, the Create and *Send *counts match. Differences in the count indicate a failure.

For example: If 50 records are successfully created and two (2) records fail, the Send stage shows 50 as Success and two (2) as Pending. The Pending count reflects the number of records that failed the Create step. If a record fails during the Send stage, it appears under the Failed column for Send.

All rows are exported into the Migration Report.

Form Submission

This step allows you to submit forms, and when appropriate, run derivation rules upon submission. During this step, attributes are not applied, ensuring greater consistency across loads and preventing concurrent asynchronous jobs.

1. Click Execute > Execute All Steps to submit forms.
2. Click Continue in the confirmation dialog.
3. Optional: Click View Details to view results in the Form Submission table.

Post-Run

The Post-Run step creates queries and executes other post-processes. It does not apply locked and frozen attributes, step 9, Freeze & Lock executes these actions.

1. Click Execute > Execute All Steps to initiate post-run processes.
2. Click Continue in the confirmation dialog.
3. Optional: Click View Details to view the Post-Run Summary page.
4. Optional: Click Log File to download a CSV of log file errors and warnings.

Reconcile Listings

This step synchronizes EDC data with the EDC UI to ensure a successful migration. Only users with System Administrator or Vault Owner security profiles can execute this step. There is no Skip option.

1. Click Execute > Execute All Steps to reconcile listings.
2. Click Continue in the confirmation dialog.

Dynamic Rules

This step executes or skips the Dynamic Rules. If it fails and rules are reset, the workflow progress table updates to reflect a Success status for the Reactivate Rules step.

1. Click Execute > Execute All Steps to activate and run Dynamic Rules.
2. Click Skip to skip this step.
3. If executing, click Continue in the confirmation dialog.
4. Optional: Click Log File to download a CSV of the dynamic rules that were executed.

Subject Statuses

With this step, Subject Status rules are either executed or skipped in the Workflow Progress table.

When executed, the system automatically sets subject statuses in the following order:

1. Pre Screen

2. Consented

3. In Screening

4. Screen Failure

5. Enrolled

6. Randomized

7. Started Treatment

8. End of Treatment

9. Withdrawn

10. Started Follow Up

11. Lost to Follow Up

12. Complete

1. Click Execute > Execute All Steps to activate and run Subject Status rules.
2. Click Skip to skip this step.
3. If executing, click Continue in the confirmation dialog.
4. Optional: Click View Details to view results and Log File to download a CSV of log file errors.

If Execute All Steps is selected in any step prior, the system attempts to execute the entire workflow. In this case, the Subject Statuses step will only work if the previous step was successful. If the previous step failed, the step will not auto-start. Once the previous step is confirmed with a status of Success, the Subject Status rules deactivated in step two are reactivated and executed.

Freeze & Lock

During Freeze & Lock, the Migrator applies frozen and locked attributes to migrating objects.

To apply frozen and locked attributes:

1. Click Execute > Execute All Steps to apply frozen and locked attributes.
2. Optional: Click Log File to download a CSV of log file errors and warnings.

Reconcile Freeze & Lock

During Reconcile Freeze & Lock, click Execute > Execute All Steps to create casebook summaries for frozen and locked objects.

Reactivate Rules

This is the final workflow step. It reactivates all rules that step 2, Deactivate Rules, deactivates.

1. Click Execute > Execute All Steps to Reactivate Rules.
2. Click Continue in the confirmation dialog.
3. Optional: Click Log File to download a CSV of the reactivated rules.

The Reset Rules after Failure option performs the same actions as the Reactivate Rules step. This option appears next to any step that fails between steps three and eight. If clicked, all the rules deactivated as part of step two are reactivated.