Execute Pilot

Pilot SOAR in your environment with your personnel to identify any tailoring, process updates, or configuration changes you’ll need to make it work smoothly. Closely monitor your pilot, identify issues, and make adjustments as efficiently as you can in order to get the most out of your pilot.

1. Execute workflows and processes
   Execute the processes and workflows that you’ve defined. Verify that they are operating correctly, and that your personnel are satisfied with their execution and their ability to monitor progress and status of operation. In particular, ensure that operations personnel can detect that a workflow is not operating correctly, and if a tool, data source, internal or external service, or piece of hardware has failed. Also ensure that engineering support personnel can troubleshoot problems effectively, and that rollback, response, and recovery procedures can be performed when required.
2. Collect metrics and measures
   Collect the metrics and measures you’ve identified, both those you routinely use to monitor operations and those specific to the pilot. Verify that you are able to collect the information that you need and that it doesn’t unduly interfere with your operations.
3. Adjust tools, workflows, processes where feasible
   As you execute your pilot, you will identify a variety of challenges, issues and shortfalls. To the extent that you are able, make adjustments to your tools, workflows, and processes to address them, so you can test them out and verify the work as anticipated. You may want even to pause your pilot for short periods of time in order to address some issues, then resume execution with the adjustments in place. You’ll want to keep track of your available resources, license restrictions, and key personnel as you’re making adjustments so you can use them all effectively.
4. Collect issues, workarounds and shortfalls
   Be sure to collect all of your issues, shortfalls and workarounds with as much detail as possible—those you make adjustments to address and also those you aren’t able to address during your pilot execution. Consider them from all perspectives: technology, personnel, processes and procedures. You’ll want to have as much information as possible to evaluate your results and to plan your initial deployment.

Helpful Information:

• If you need to adjust your processes or workflows, you may want to refer back to the Introduction to IACD Playbooks and How to Build an IACD Playbook whitepapers and the Playbook Thin Specification, and the additional information and tutorials found in the IACD Playbooks and Workflows section.
• For suggestions from experiences from other SOAR adopters, take a look at Implementer Insights, Operationalization Lessons Learned, and the IACD & FS-ISAC Financial Pilot Results.
• You may also want to revisit the orchestration examples in the Orchestration Example: Automated IT/OT Recovery and Advanced Orchestration Techniques: Reversibility video.
• You may also want to revisit metrics and measures, described in Security Automation and Orchestration Metrics and Measures.

Evaluate Pilot Results

When you’ve completed your pilot, evaluate what you’ve learned. You can usually start this evaluation while you’re still executing your pilot, but you’ll want to step back and fully evaluate after all the results are in. Did you learn what you needed to learn? Can you fix any issues you identified? Based on what you’ve learned in your pilot, you can better define everything you’ll need to proceed to fully operational SOAR.

1. Evaluate metrics and success measures
   Evaluate the metrics and other measures of success you defined for your pilot. Were you able to measure what you wanted to measure? Do those measurements actually align with the metrics you want to evaluate? Were you as successful as you’d hoped? Where did your implementation fall short? Did you test adjustments and workarounds? How effectively did they address the issues you found? What issues and shortfalls are still present? Can you fix them? Can you develop reasonable workarounds?
2. Confirm satisfaction with organizational policy and procedure modifications
   You’ll be making modifications to your organizational policy and procedures. You'll want to confirm that your high-priority processes and workflows were satisfied, and make any adjustments accordingly. When these were in place during the pilot, what issues were created? What workarounds were performed? How effective were they? Did anyone feel that policies or procedures “got in the way”? What solutions or workarounds were proposed? The ICD Conceptual Reference Model (Reference 1) can help you keep your changes aligned with your overall SOAR architecture and playbook guidance (Reference 3) can help you update your playbooks and workflows.
3. Identify viable processes and workflows
   Once you’ve evaluated your metrics and success measures and confirmed your policy and procedure modifications, do you have one or more viable processes, playbooks and associated workflows? Did they perform effectively for your needs? Were you able to detect failures and reasonably recover from them? Do their resource, support, and personnel requirements fall within the constraints you have?
4. Identify costs (resources, personnel, training) and timeline to make desired workflows viable
   What will you need to roll out these processes and workflows to your organization? What hosting and support resources do they require? What are the licensing costs for your larger organization? What skillsets do your personnel need to have and how many people will you need with each skillset? How long will it take to create training packages? How long will it take to purchase hardware? How long to train your personnel? You may find that some of what you piloted isn’t quite mature enough yet—what will it take to make it viable?

Helpful Information:

• For information to help evaluate your metrics and measures, take a look at Security Automation and Orchestration Metrics and Measures.
• For suggestions from experiences from other SOAR adopters, take a look at Implementer Insights, Operationalization Lessons Learned, and the IACD & FS-ISAC Financial Pilot Results.
• If your pilot included Indicators of Compromise (IOCs), you may find the Actionable Information Sharing overview helpful.

Select Focus for Initial Deployment

After you’ve evaluated all your results, you can decide what you’re really ready to deploy. It’s generally best to be a little conservative and carefully scope what you’re going to initially deploy. You may want to consider one or more interim operational states to start—you can incrementally add more once you’ve familiarized all of your personnel with SOAR and identified how it fits into your organization.

1. Identify supportable processes and workflows
   When you’ve evaluated the results of your pilot and the costs and timeline needed to establish your processes and workflows, you can decide which ones are ready to roll out for initial deployment. You should not only consider whether your processes and workflows work correctly, but also that they can be supported at your full operational scale. Do you have enough storage for information? Do you have enough processing power (both human and computer)? Do you have sufficient bandwidth to transfer information at operational scale?
2. Evaluate resource requirements (personnel, equipment, facilities)
   Next, carefully evaluate what you’ll need to deploy these processes and workflows. Which ones do you have the personnel to support (both ops and engineering)? Your new automated processes will generate information and alerts at a different rate than your previous ones. You will likely need to allocate your personnel differently, and they may need to use different skillsets. Do you have the resources to purchase, license and provision any additional hardware or software you’ll need? To train all your personnel? How long will it take to get everything ready?
3. Ensure “failsafe” mechanisms are available
   There will inevitably be glitches as you are rolling out your new processes and workflows. If you’ve prepared for that, you can minimize the negative impact to your operations. Be sure you’ve made provisions so your workflows can fail safely, and that you can efficiently detect and recover from any errors or mistakes.

Helpful Information:

• For evaluating the supportability and resource requirements of your piloted processes and workflows, you may want to refer back to the Introduction to IACD Playbooks and How to Build an IACD Playbook whitepapers and the Playbook Thin Specification, and the additional information and tutorials found in the IACD Playbooks and Workflows section. You may also want to revisit the High-Benefit/Low-Regret Automated Actions as Common Practice whitepaper.
• The S-PET tool, including the Product Integration tab, can help you evaluate and keep track of resource requirements, including interoperability with other tools and equipment.
• For suggestions from experiences from other SOAR adopters, take a look at Implementer Insights, Operationalization Lessons Learned, and the IACD & FS-ISAC Financial Pilot Results.
• The orchestration examples in the Orchestration Example: Automated IT/OT Recovery and Advanced Orchestration Techniques: Reversibility videos may provide insight and considerations for full operations scale.
• You may also want to more broadly incorporate metrics and measures described in Security Automation and Orchestration Metrics and Measures.
• If your candidate process(es) includes Indicators of Compromise (IOCs), the Information Sharing under IACD page as well as the Actionable Information Sharing overview may also be helpful for determining process/workflow supportability.

Evaluate and Acquire Resources

Next, plan out your resources. You probably already have plans to acquire and upgrade your IT infrastructure and operational resources. You’ll want to take advantage of any plans you already have in place, and not disrupt any that you don’t need to. You’ll also want to be sure to have everything you’ll need, including secondary tools and equipment as well as resources for monitoring the health, status and security of your SOAR solutions.

1. Analyze current and planned IT resources and infrastructure
   Take a look at your current resources and infrastructure. What current assets can host your initial deployment? What planned assets can you use? Will they be available on the timeline you need? Do they need any upgrades? Do they have the security provisioning and approvals you’ll need? Will you be repurposing any assets?
2. Obtain or update tools, plug-ins, equipment
   Now take a look at the tools, plug-ins, and equipment you’ll need for your fully integrated solution. Do any need upgrades? Do you need additional licenses? Do you need any special equipment or tools (e.g., for transport or storage of sensitive data)? You can use the S-PET (Reference 2) to keep track and evaluate options.
3. Include resources for health/status and security monitoring (equipment and personnel)
   Also consider how you will monitor your SOAR solution. How are you checking that your tools and data sources are up and running? What are your access control or data protection needs? How will you assure they are not compromised? Are your tools performing the actions they are supposed to take (and no others)? You may need additional tools, licenses or workstations for monitoring operations.

Helpful Information:

• The S-PET tool, including the Product Integration tab, can help you evaluate and keep track of your tool requirements, including interoperability with other tools and equipment.
• The ICD Conceptual Reference Model and IACD Baseline Architecture whitepapers can help you identify interoperability requirements and maintain consistency with your broader SOAR architecture,
• For suggestions from experiences from other SOAR adopters, take a look at Implementer Insights, Operationalization Lessons Learned, and the IACD & FS-ISAC Financial Pilot Results.
• If your candidate process(es) includes Indicators of Compromise (IOCs), the Actionable Information Sharing overview and the AIS Fact Sheet provide more details to help identify interoperability and resource requirements.

Prepare Environment and Personnel

Finally, set everything up.

1. Update operational policy and procedures
   Based on what you learned in your pilot and the processes, playbooks and workflows you’ve selected for your initial deployment, you’ll likely need to make some final changes to your operational policy and procedures. Doing that now will help to keep your activities aligned throughout your deployment.
2. Update training packages and train operations and support personnel
   You’ll now be training all of your personnel, both in operations and engineering support. While this training is focused on initial deployment, you will be reusing most, if not all, of your training packages to on-board new personnel in the future. Some people may only need an awareness of the overall changes, including policy and procedures, that can be accomplished during a lunchtime brownbag. Some will require specific training for new tools or procedures, and some may require more extensive or specialized training. Think through the training that will be needed, and how to provide it most efficiently. Be sure not to overlook information and training available from your SOAR solution providers or from the IACD website. Where hands-on training is needed, also include the tools and data you will need for training.
3. Install and test operational equipment, tools, and workflows
   Before going live, you’ll need to install and test all your tools and equipment to ensure it’s all configured properly. Don’t forget to check out non-normal operations such as system/tool failure and recovery processes, too. Organizations typically set up a staging or test environment that mimics their operational environment as closely as possible. You’ll probably find that you want to maintain this environment to deploy new upgrades efficiently.
4. Allocate engineering support
   Sufficient engineering support is critical for SOAR solutions, especially during initial deployment. Depending on the scope of your initial deployed solution, it’s likely you’ll need additional engineering support for a few days or weeks when you first go live. There will inevitably be glitches, but there will also be additional support requests as the organization adjusts to new tools, processes and modes of operation. It may also take longer to resolve support requests until your support personnel are sufficiently familiar with the new environment. Your SOAR tool vendors may provide additional support
5. Develop transition plan
   As you migrate processes and capabilities to SOAR solutions, it is likely you will repurpose or retire existing hardware, tools, and data. You’ll need a transition plan to be sure to minimize disruption to your operations and to comply with all relevant policies. What information needs to be migrated or archived? Be sure to consider security and policy auditing requirements, in addition to current operational needs for historical information. Does information or hardware need to be sanitized before migrating or repurposing? What system services or capabilities need to be migrated or retired? Who needs to be informed and how much advance notification do they need? How will information be recovered or services temporarily restored if necessary?

Helpful Information:

• Refer back to your tools and considerations captured in the S-PET tool, including the Product Integration tab, to help you keep track of your tool requirements, including interoperability with other tools and equipment.
• The ICD Conceptual Reference Model and IACD Baseline Architecture whitepapers can help you maintain consistency with your broader SOAR architecture,
• For suggestions from experiences from other SOAR adopters, take a look at Implementer Insights, Operationalization Lessons Learned, and the IACD & FS-ISAC Financial Pilot Results.
• And don't forget to pay close attention to your metrics and measures (Security Automation and Orchestration Metrics and Measures).

Ready to Initially Deploy

Now you're ready to initially deploy SA&O. You should have the following:

1. All personnel allocated and trained
2. Hosting and support resources acquired
3. Operating procedures defined
4. Transition Plan Prepared

At a minimum, your plan for deployment should include:

• Operational modes and constraints
  • Relationship to current operations (interfaces, data and tools)
  • Availability and downtime requirements (consider tool mainteance windows)
• Personnel
  • Required skills, expertise and training
  • Appropriate personnel identified and allocated
  • Management reporting and decision-making requirements identified and documented
  • Personnel training schedule
• Operational process and procedure modifications
  • Metrics determined and measurement points identified
  • Security requirements developed and approved
  • Modifications to current systems and processed identified and implemented
  • "failsafe" mechanisms identified and verified to be working
• Processes and Procedures for
  • Orchestration playbooks identified and baselined
  • Operational workflows developed
  • Risk management and service restoration procedures defined (e.g., backup/recovery, fallback/rollback, etc.)
  • Support procedures and resources defined and verified (e.g., technical support contact lists, anticipated O&M remote access, call in lists, etc.)
• Systems and Tools
  • Products, tools and equipment updated and baselined, with license numbers and expiration dates documented
  • Database and backup synchronization defined and verified
  • Processing, memory and storage capacity (including surge capacity or 'operating margin') estimated, allocated and monitored for all clients and servers
  • Monitoring of SA&O and affected systems documented and verified (displays, status indicators, critical thresholds/limits identified, etc.)
  • System resiliency and restoration capabilities (e.g., failover) documented and verified
• Interfaces
  • IT System Interfaces (sensors, tools, monitoring, etc.)
  • Input data (e.g., threat/reputation feeds)
  • Sharing/output data

In addition, you should have a transition plan that defines additional measures necessary to migrate or decommission assets, personnel and software currently in operational use as required to support new SOAR operations. This transition plan should include the following:

• Transition Plan Overview
  • Required systems needs, including data needs
  • Incorporation into the current architecture
  • Any required capability, tool or data migration
  • Security and privacy consideration
  • Method/schedule to transition users from the old system or tool to new
  • Method/schedule to decommission the old system
• Approval and Notification
  • Method and schedule for system/tool owners to review and approve transition plan
  • Method and schedule for identification and notification of affected users, with provision for issue identification, feedback and resolution
• Replacement System, Capability, or Service
  • Activities and schedule for replacement system to be configured and tested
  • Activities and schedule for replacement tools, capabilities or services to be installed and tested
  • Activities and schedule for data to be migrated and verified
  • Persistent references for procedures, data, backups, archives, and documentation (e.g., user’s guides or operations manuals) sufficient for temporary system, capability or service re-initiation if necessary
• Information retention
  • Relevant policy references for the archiving and retention of information, including retention requirements and constraints (e.g., time, format, media type or size)
• Configuration Management Information
  • Engineering documents and source code including hardware/software version and licensing information, installation guides and operating procedures, interface documentation, user’s guides, configuration files, system and network drawings